It's a matter of least-harm. Keeping magic away from Muggle view is the best way. When that fails, a minor deletion of memory is a lesser but acceptable option.

What other options are there? Killing the Muggle? Kidnapping them and never letting them have contact with 'normal' human society? Removing the memory only of the incident isn't a bad alternative.

Yes, the technique could certainly be abused. But the way it's being applied isn't particularly objectionable. It certainly shouldn't be the first line of defense, but it isn't being treated that way.