I've got it set at 500 comments, but I don't think it actually shows 500-- and in any case, I think it's just for comment threads, not for recent comments.

I never noticed that before, but you are right: all the /comments/ pages I have asked for have 100 comments on them regardless of how I try to change that. (I tried setting the number in prefs to a smaller value, logging out and in again, following a "Next" link.)

(Oddly, although it will show me a page with 100 comments on it if I click it, the URL in the "Next" link bottom of a /comments/ page contains the string "count=50".)