Surely if you provably know what the ideal FAI would do in many situations, a giant step forward has been made in FAI theory?

BBC Radio : Should we be frightened of intelligent computers? http://www.bbc.co.uk/programmes/p01rqkp4 Includes Nick Bostrom from about halfway through.

Drat. I just came here to post that. Still, at least this time I only missed by hours.

You need a different definition for 'blackmail' then. Action X might be beneficial to the blackmailer rather than negative in value and still be blackmail.

Why not taboo 'blackmail'? That word already has a bunch of different meanings in law and common usage.

Omega gives you a choice of either $1 or $X, where X is either 2 or 100?

It seems like you must have meant something else, but I can't figure it out.

Isn't that steel-man, rather than strong-man?

Reading that, I thought: "I bet people asking questions like that is why 'Original Sin' got invented".

Of course, the next step is to ask: "Why doesn't the priest drown the baby in the baptismal font, now that its Original Sin is forgiven?"

…

I, Robin, or Michael Vassar could probably think for five minutes and name five major probable-big-win meta-level improvements that society isn't investing in

Are there lists like this about? I think I'd like to read about that sort of stuff.

I remember seeing a few AI(and other things, sometimes) debates (mostly on YouTube) where they'd just be getting to the point where they are clarifying what it is that each one actually believes and you get: 'agree to disagree'. The end.

Just when the really interesting part seemed to be approaching! :(

For text-based discussions that fail to go anywhere, that brings to mind the 'talking past each other' you mention or 'appears to be deliberately misinterpreting the other person'