It's hard to know the difference between "I don't have visual mental imagery" and "I'm overly negative about what my mental imagery looks like", of course. The three things that most strongly lead me to believe I don't have visual mental imagery are

• the huge difference between what I see when nearly asleep and what I see normally

• descriptions of mental imagery differences and changes like cousin_it's aural imagination and fburnaby's note and e.g. this passage from Galton's paper:

  To my astonishment, I found that the great majority of the men of science to whom I first applied, protested that mental imagery was unknown to them, and they looked on me as fanciful and fantastic in supposing that the words 'mental imagery' really expressed what I believed everybody supposed them to mean. They had no more notion of its true nature than a colour-blind man who has not discerned his defect has of the nature of colour. They had a mental deficiency of which they were unaware, and naturally enough supposed that those who were normally endowed, were romancing. To illustrate their mental attitude it will be sufficient to quote a few lines from the letter of one of my correspondents, who writes:--

  "These questions presuppose assent to some sort of a proposition regarding the 'mind's eye' and the 'images' which it sees….. This points to some initial fallacy…… It is only by a figure of speech that I can describe my recollection of a scene as a 'mental image' which I can 'see' with my 'mind's eye'….. I do not see it… any more than a man sees the thousand lines of Sophocles which under due pressure he is ready to repeat. The memory possesses it, &c."

• discussions with those who claim they do have visual mental imagery, and their incredulity about my descriptions of my experience - incredulity that does not feel like they would simply describe their own experience differently. My sister, for example, is a writer, and when I described my lack of visual mental imagery said that she finally understood why I didn't seem to understand the beauty of certain prose she'd shown me, because its beauty was mainly in the images it inspired.