I don't even know how to answer this because it's coming from a place that's so foreign to me. I have a quite weak visual imagination (not full-on aphantasia), and I've never heard a voice speaking words in my head when I read (although actually, now that I've listened to a lot of audiobooks, I can force this to happen briefly if I concentrate). But I've always enjoyed reading! To me, I guess I would say, words are just sort of fundamental? Like, the word itself, the shape of squiggles on the page, is the thing that has meaning, and I don't have to visualize anything beyond it to understand it. It's like the difference between reading a foreign language you're not that good at, where you still have to translate every thought into your native language to really understand it, versus reading in your native language, where there's no translation step.
It might be true, as quoted in Mo's comment below, that people with weak visual imaginations are less likely to enjoy extremely visual-description-heavy fiction like Lord of the Rings – I indeed found reading the LotR books mind-numbingly boring, borderline painful. But in almost all works there's a ton of content that can be enjoyed without imagining it into being: concepts, emotions, even most kinds of events. I also find myself very drawn to beautiful writing, wordplay, and just skillful use of language in general, and now that I think about it, this is probably why. Neat!
The same applies to nonfiction – building an understanding of the connections between concepts doesn't have to rest on any sort of visual framework. The concepts can just become connected, literally/physically, in the structure of your brain.
I do think that I would have struggled less with university-level physics and the related math if I had more of a visual imagination. It's quite hard to keep track of things when it all just feels like symbols you're manipulating; I imagine that being able to visualize things would have let me viscerally understand connection between the math and the physical systems being described. As it was I just sort of, knew explicitly that they were connected. But it was all very vague and confusing.
I hope this helps you understand the experience somewhat. Feel free to ask me followup questions.
I have not tried the square test before, and it's weird. At my first attempt I just completely failed. I've certainly seen enough squares in my life to imagine them, but it just did not happen. Then I imagined drawing that square - not the tactile sensition of it, but just the process of going from A to B to C to A, but that only gets me the 3rd type of square. I can push it to the 4 with additional effort, but I can't seem to get past that just yet. So it's far from red.
The shape is certainly easier for me to imagine than color, colors tend to be really bleak.
It reminds me of another classic example, where they ask you to imagine an apple.
At my very first attempt I found that difficult for some reason, but after a while I have no trouble imagining any apples I want - green, red, yellow, mixed color, stem with leaf or without leaf, no stem, partially eaten, cut in half, partially rotten, with a worm inside it, etc etc.
But then again I have a lot more experience paying attention to apples then to abstract red squares, even if I do see squares way more often. Maybe it adds to effect. Or maybe all the possible transformations of shape distract me enough from color so that I fail to notice how poor my imagination of it really is.