It may simply be that the ugly and dumb people go unnoticed.

Absolutely. The stereotype of the smart geek/nerd comes from the fact that when people are ugly/socially awkward/weird, other people get positively surprised that they are smart and really notice that. It is like, they would pretty much "written them off" as low-status unimportant people to be ignored, and thus they get surprised that they actually have useful virtues, and should not be so easily ignored because while how they say things is not popular, what they say is often true and insightful.

While the dumb nerd/geek just gets ignored forever.