An FAI would have some security advantages. It can achieve physical security by taking over the world and virtualizing everyone else
That is your exact wording. Not "In the event that the AGI determines that it's safe to [euphemism for doing something that could mean killing the entire human race] because there are software copies." or "if virtualizing is safe..."
Even if your wording was that, I'd still disagree with it.
I thought the most important reason to do friendliness research was to give the AGI what it needs to avoid making decisions that could kill all of humanity. It is humanity's responsibility to dictate what should happen in this case and ensure that the AGI understands enough to choose the option we dictate. If you aren't in favor of micromanaging the millions of tiny ethical decisions it will have to make like exactly how many months to put a lawbreaker in jail, that's one thing. If you aren't in favor of making sure it decides correctly on issues that could kill all of humanity, that's negligent beyond imagining. If you are aware of a decision that an AGI could make that could kill all of humanity, and you are in favor of creating an AGI that hasn't been given guidance on that issue, then you're in favor of creating a very dangerous AGI.
Advocating for an AGI that will kill all of humanity vs. advocating for an AGI that could kill all of humanity is a variation on "advocating violence" (it's advocating possible violence) but, to me, it's no different from saying: "I'm going to put one bullet in my gun, aim at so-and-so, and pull the trigger!" - Just because the likelihood of killing so-and-so is reduced to 1 in 6 from what's more or less a certainty does not mean it's not a murder threat.
Likewise, adding the word "possibly" into a sentence that would otherwise break the censorship policy is a cheap way of trying to get through the filter. That should not work. "We should possibly go on a killing rampage." - no.
What's most alarming is that you've done work for SIAI.
The whole point of SIAI is not to go "Let's let the AGI decide what is ethical" but "Let's iron out all the ethical problems before making an AGI!"
If Eliezer doesn't want to look bad, he should consider this.
As I clarified in a subsequent comment in that thread, "if the FAI concludes that replacing a physical person with a software copy isn't a harmless operation, it could instead keep physical humans around and place them into virtual environments Matrix-style."
We could argue about whether to build an FAI that can make this kind of decision on its own, but I had no intention of doing anyone any harm. Yes the attempted-FAI may reach this conclusion erroneously and end up killing everyone, but then any method of building an FAI has the possibility of ...
New proposed censorship policy:
Any post or comment which advocates or 'asks about' violence against sufficiently identifiable real people or groups (as opposed to aliens or hypothetical people on trolley tracks) may be deleted, along with replies that also contain the info necessary to visualize violence against real people.
Reason: Talking about such violence makes that violence more probable, and makes LW look bad; and numerous message boards across the Earth censor discussion of various subtypes of proposed criminal activity without anything bad happening to them.
More generally: Posts or comments advocating or 'asking about' violation of laws that are actually enforced against middle-class people (e.g., kidnapping, not anti-marijuana laws) may at the admins' option be censored on the grounds that it makes LW look bad and that anyone talking about a proposed crime on the Internet fails forever as a criminal (i.e., even if a proposed conspiratorial crime were in fact good, there would still be net negative expected utility from talking about it on the Internet; if it's a bad idea, promoting it conceptually by discussing it is also a bad idea; therefore and in full generality this is a low-value form of discussion).
This is not a poll, but I am asking in advance if anyone has non-obvious consequences they want to point out or policy considerations they would like to raise. In other words, the form of this discussion is not 'Do you like this?' - you probably have a different cost function from people who are held responsible for how LW looks as a whole - but rather, 'Are there any predictable consequences we didn't think of that you would like to point out, and possibly bet on with us if there's a good way to settle the bet?'
Yes, a post of this type was just recently made. I will not link to it, since this censorship policy implies that it will shortly be deleted, and reproducing the info necessary to say who was hypothetically targeted and why would be against the policy.