I use a beard trimmer and take it back to about an eighth of an inch whenever someone gets annoyed by it.

The actual length will vary for you: you should experiment until you find a length that is just longer than "sandpaper feel". But certainly, your argument is solid: there's no point going for the closest possible shave when just a close shave will do; and for me, there's no point going for a close shave when just a trim will do.