I am the same way, I have a freaky memory and my math skills were a little lacking. I said fuck it and went into mechanical engineering anyways. I'm really glad I did and I'm now a senior, so I'm almost done (3.4 gpa). Don't let the math scare you away, you'll pick it up along the way. The math I took was Calc I through Calc III and differential equations. Tough stuff, but not impossible. Much of it is multi-step calculations where your memory will be a huge asset.

EDIT: I use algebra and physics more than anything else. The only calculus I use is differentiation and integration, and most of that is pretty basic.