Why don't we see many more people wearing red?

I was told when I was younger that red didn't look good on me because of my pink Irish skin.

Can the rest of the color spectrum be ranked similarly in its effect on referees' evaluations?

Not a lot of evidence here. Some psychologists briefly thought blue had an advantage over white in judo matches but it turned out that combatants in blue were higher seeds and thus were expected to perform better. Black seems to have some similar effects of red but I don't think they've ever been paired head-to-head.

Yellow got it's ass kicked in English football. Other than that I don't know.