Would it be legitimate for me to downvote people the next time they mention ideas that I consider completely crackpot (like quantum immortality), or ideas that most of the world considers completely crackpot (like many-worlds)?

I tend to presume less control of other people's voting behavior than you seem to.