I think that's a fallacy; humans aren't good at adding up large numbers of small utilities. But by your logic e.g. the "salami-slice fraud" (stealing 0.1 cent from everyone on the planet) would be ethical - it increases your own happiness, and has no effect on everyone else.
If it really had absolutely no effect, then I guess our moral duty would be to steal that money and give it to efficient charity.
Just because humans are not good at observing something, that does not mean it doesn't exist. Sure, in real life, the effects of losing 0.1 cent are invisible to humans, and probably the observation itself would be more costly than the 0.1 cent. But do it repeatedly, and the effects start being observable. Also, there is a problem of transaction costs.
Utilitarianism seems to indicate that the greatest good for the most people generally revolves around their feelings. A person feeling happy and confident is a desired state, a person in pain and misery is undesirable.
But what about taking selfish actions that hurt another person's feelings? If I'm in a relationship and breaking up with her would hurt her feelings, does that mean I have a moral obligation to stay with her? If I have an employee who is well-meaning but isn't working out, am I morally allowed to fire him? Or what about at a club? A guy is talking to a woman, and she's ready to go home with him. I could socially tool him and take her home myself, but doing so would cause him greater unhappiness than I would have felt if I'd left them alone.
In a nutshell, does utilitarianism state that I am morally obliged to curb my selfish desires so that other people can be happy?