Is it just me, or is non-consensual sex obviously a bad thing?

"Obviously bad" isn't a utilitarian justification.

Banning Dalits from going within 96 feet of Namboothiris has much more harm done to Dalits than Namboothiris' feelings of ritual pollution. This isn't the case with non-consensual sex.

To play the Devil's advocate:

1. I expect you seriously underestimate the strength of Namboothiris' feelings. To us it seems like pure religious madness, moreover we feel outrage at the extreme inequality existing because of ancient caste prejudices, so we tend to sympathise with the Untouchables and regard the traditional Brahmin rights as unjust. But it doesn't seem that way from the Brahmin perspective.
2. Some of the unpleasantness connected with non-consensual sex is probably status related - being raped makes one lose a lot of status and we tend to avoid status loss. I wonder how much less serious problem would rape become in a society where the negative status effects were removed. We find it acceptable to solve the caste problem by rebuilding the society and changing the people's values - even when many people are objecting; why not attempt the same approach to rape?

(Disclaimer: I think that caste society is unjust and I don't actually wish to change our society to be more rape-tolerant. But I am no utilitarian. This comment is a warning against creating fake utilitarian explanations of moral judgements made on non-utilitarian grounds.)