It's really rather confusing and painful to have a discussion without talking about what you are discussion, using only veiled words, indirect hints, and private messages. May I suggest that instead we move the discussion to the uncensored thread if we are going to discuss it at all?
I believe that the whole point of the uncensored thread is to provide a forum for the discussion of topics, such as basilisks, that Eliezer is uncomfortable with us discussing on the main site.
I don't know of anyone who's seriously thinking about this particular problem and would want to discuss it there. Maybe someone will prove me wrong, though.
I think I've found a new argument, which I'll call X, against Paul Christiano's "indirect normativity" approach to FAI goals. I just discussed X with Paul, who agreed that it's serious.
This post won't describe X in detail because it's based on basilisks, which are a forbidden topic on LW, and I respect Eliezer's requests despite sometimes disagreeing with them. If you understand Paul's idea and understand basilisks, figuring out X should take you about five minutes (there's only one obvious way to combine the two ideas), so you might as well do it now. If you decide to discuss X here, please try to follow the spirit of LW policy.
In conclusion, I'd like to ask Eliezer to rethink his position on secrecy. If more LWers understood basilisks, somebody might have come up with X earlier.