There are different degrees of severity. Being perceived as weird in a nerdy way is low-status, but it's nothing compared to being perceived as harboring fundamentally evil views. Most notably, the former sort of low status isn't infectious; you can associate with weird nerdy people without any consequence for the other aspects of your life. Not so when it comes to associating with the latter sort of people.