CEV is a construct for AI purposes that actual human beings can't eval - I don't think I've ever seen a human discussion that was helped by invoking it. It's not like Solomonoff Induction where sometimes you really can be helped by thinking formally about Occam's Razor. In practice, human beings arguing about ethics are either already approximating their part of the 'good' as best they can, or they're confused about something much simpler than CEV, like consequentialism. If you should never use the word 'truth' when you can just talk about the object level, and never say 'because it's not optimal!' when that just means 'I don't think you should do that', then there's basically never a good time to talk about CEV - it always deflates out of the sentence unless you're talking directly about FAI.
It's not like Solomonoff Induction where sometimes you really can be helped by thinking formally about Occam's Razor. In practice, human beings arguing about ethics are either already approximating their part of the 'good' as best they can, or they're confused about something much simpler than CEV, like consequentialism.
It's exactly like Solomonoff Induction where most of the time you really can't be helped by thinking formally about Occam's Razor. In practice, human beings arguing about probabilities are either already approximating their part of the 'simple' as best they can, or they're confused about something much simpler (haha) than Solomonoff Induction, like Bayesianism.
r/HPMOR readers on heroic responsibility - not the OP, the comments. Holy snorkels this is good.