FYI, you're mostly right, at least based on my experience. I tend to have a much harder time listening to Audiobooks of SF/fantasy, and a harder time listening to any fiction vs. non-fiction.

I also have a much easier time listening to SF/Fantasy when it's in a setting I already know (e.g. sequels, books I've read before, etc). Also easier to listen to books from authors I read a lot (but that may be true in general, come to think of it).

I still highly recommend anyone who can to listen to Audiobooks, at least of non-fiction, as one of the best and easiest hacks around.