There is a meaningful distinction between believing that utility should be agent neutral and believing that it should be agent relative. I tend to assume people are advocating an agent neutral utility function when they call themselves utilitarian since as you point out it is rather a useless distinction otherwise. What terminology do you use to reflect this distinction if not utilitarian/non-utilitarian?

It's the agent neutral utilitarians that I think are dangerous and wrong. The other kind (if you want to still call them utilitarians) are just saying the best way to maximize utility is to maximize utility which I have a hard time arguing with.