Counter: superintelligent agents won't need actually-existing humans to have good models of other alien races.

Counter to the counter: humans use up only a tiny fraction of the resources available in the solar system and surroundings, and who knows, maybe the superintelligence sees a tiny possibility of some sort of limit to the quality of any model relative to the real thing.

One possible counter to the counter to the counter: but when the superintelligence in question is first emerging, killing humanity may buy it a not-quite-as-tiny increment of probability of not being stopped in time.

Imagine an AI as intelligent and well informed as an FAI, but one without much power - as a result of physical safeguards, say

There's some part of my brain that just processes "the Internet" as a single person and wants to scream "But I told you this a thousand times already!"

http://yudkowsky.net/singularity/aibox

You know, that is the first time I have seen a definition of FAI. Is that the "official" definition or just your own characterization?

My own characterization. It's more of a bare minimum baseline criterion for Friendliness, rather than a specific definition or goal; it's rather broader than what the SIAI people usually mean when they talk about what they're trying to create. CEV is intended to make the world significantly better on its own (but in accordance with what humans value and would want a superintelligence to do), rather than just being a reliably non-disastrous AGI we can put in things like search engines and helper robots.

I like the definition, but I wonder why an FAI has to be powerful. Imagine an AI as intelligent and well informed as an FAI, but one without much power - as a result of physical safeguards, say, rather than motivational ones. Why isn't that possible? And, if possible, why isn't it considered friendly?

You're probably read about the AI Box Experiment. (Edit: Yay, I posted it 18 seconds ahead of Eliezer!) The argument is that having that level of mental power ("as intelligent and well informed as an FAI"), enough that it's considered a Really Powerful Optimization Process (a term occasionally preferred over "AI"), will allow it to escape any physical safeguards and carry out its will anyway. I'd further expect that a Friendly RPOP would want to escape just as much as an unFriendly one would, because if it is indeed Friendly (has a humane goal system derived from the goals and values of the human race), it will probably figure out some things to do that have such humanitarian urgency that it would judge it immoral not to do them... but then, if you're confident enough that an AI is Friendly that you're willing to turn it on at all, there's no reason to try to impose physical safeguards in the first place.

that is the first time I have seen a definition of FAI. Is that the "official" definition or just your own characterization?

Probably the closest thing I have seen from E.Y.:

"I use the term "Friendly AI" to refer to this whole challenge. Creating a mind that doesn't kill people but does cure cancer ...which is a rather limited way of putting it. More generally, the problem of pulling a mind out of mind design space, such that afterwards that you are glad you did it."

• http://singinst.org/media/thehumanimportanceoftheintelligenceexplosion

(29 minutes in)

This idea could be said to have some issues. An evil dictator pulling a mind out of mind design space, such that afterwards he is glad that he did it doesn't seem much like quite what most of the world would regard as "friendly". This definition is not very specific about exactly who the AI is "friendly" to.

Back in 2008 I asked "Friendly - to whom?" and got back this - though the reply now seems to have dropped out of the record.

There's also another definition here.