I firmly believe the problem with math education is they move to rapidly from physicaly reality (1 stick + 1 stick = 2 sticks) to abstraction. A^2 + B^2 = C^2. If they would teach, for example, the pythagorean theorum but showing you how the squares outside the triangle add up, kids would have a much easier time learning the abstracted algrebra behind it. It's a sin they steal geometry from our children.
Yeah, it could be to do with an inability to visualise things that makes it difficult to process and understand.
Obviously some people are just smarter than other people, but I do think you can cultivate spacial reasoning by literally just thinking (i.e. playing) in 3d space with physical objects.