A great point. I can confidently say mine is at least "average", likely above average. I consider myself a "visual learner," with good "story memory" and I agree that as such memory palaces are a particularly good for me. However, when I use the technique, I'd say it's mostly non-visual. I'd guess it's 20% me "seeing" the room, 10% "everything else" (texure, sound, smell, emotion, all of which I find much harder but make deliberate effort to employ), and 70% conceptual "The spaceship is crashing through the door, sending shards of wood scattered across the bedroom". That is one of the many "secrets" that make the technique so useful to me: most every object should perform an action that would in real life permanently damage or alter the room. With tricks like that, I think it is helpful for most all people, even those not visually inclined.

For what it's worth: I have no data on this myself, but my study coach posits that everyone can do it, some have more trouble than others, but when done well it's so effective that most everyone should benefit. He says most of his students are resistant, but almost all of them profess loving it once they develop the skillset to use it.

My local LW group did some exercises with them and they worked rather well. I don't have a very good ability to internally visualize but was still able to remember significantly more with a memory palace.

Find a friend to try it a couple of times and see how it works for you.