But I don't think a comment has to directly serve my terminal values to be worth upvoting.

Agreed, and I may be overbroadening "terminal value" by applying it as "things worth upvoting." A comment being "nice" makes it more likely that comment will be "helpful," and I think "helpful" comments are worth upvoting; do I think comments that are nice but not helpful are worth upvoting? Not really, and a policy to upvote for niceness alone won't capture that value judgment.

But it could be that niceness is the best cause or proxy for desired consequence, and thus is worthwhile.