Actually, IIRC, Eliezer said that he thinks Robin Hanson's (I think it was his) solution to the mugging seems to be in the right direction. But that gets into computational power issues. Actually, my original intent was to name this sequence "How not to Be Stupid (given unbounded computational power)"

Obviously we can't do the full decision theory computations in full exact correctness. And I did give the warning against hastily giving an oversimplified human preference generator. What I'm going for here is more "why assume that Bayesian decision theory is the thing we should be building approximations to, rather than some other entirely different blob of math?"

(Oh, incidentally. I originally chose SPECKS, then later one of the comments in that sequence of posts (the comment that stepped through it, incrementally reducing, etc) ended up convincing me to switch to TORTURE.)

Also, finished editing the offending argument.