there's a difference between thinking briefly and abstractly of the idea of something and indulging in fantasy about it.

Yes, of course, there is a whole range of, let's say, involvement in these thoughts. But if I understand mainstream Catholicism correctly, even a brief lustful glance at the neighbor's wife is a sin. Granted, a lesser sin than constructing a whole porn movie in your head, but still a sin.