I agree there still would be very easy ways to punish enemies or even more common 'friends' that don't toe the line.

I do think it would identify some interesting cliques or color teams. The way I envision using it would be more topic category based. For instance, for topic X I average this group of peoples opinions but a different group on topic Y.

On the positive side, if you have a minority position on some topic that now would be downvoted heavily you could still get good feedback from your own minority clique.