well, it could be and often is harmful to someone, if and only if you act upon it.

The thought itself is an object in reality, and you can care about objects you can't observe. If your though itself implements a tortured person, you shouldn't think that thought, even if there is no possibility of somehow "acting" on it, even if thinking that thought improves your actions according to the same human moral reference frame. This is not as extreme for mere human thought, but I see no reason for the thoughts in themselves to be exactly morally neutral (even if they count for very little).