Good point, but my question was about what we can do to raise chances that it will be friendly AI.

Nothing, because we still don't know what a friendly AI is.

We live in an increasingly globalised world, where moving between countries is both easier in terms of transport costs and more socially acceptable. Once translation reaches near-human levels, language barriers will be far less of a problem. I'm wondering to what extent evaporative cooling might happen to countries, both in terms of values and economically.

I read that France and Greece lost 3 & 5% of their millionaires last year (or possibly the year before), citing economic depression and rising racial/religious tension, with the most popular destination being Australia (as it has the 1st or 2nd highest HDI in the world). 3-5% may not seem like a lot, but if it were sustained for several years it quickly piles up. The feedback effects are obvious - the wealthier members of society find it easier to leave and perhaps have more of a motive to leave an economic collapse, which decreases tax revenue, which increases collapse etc. On the flip side, Australia attracts these people and its economy grows more making it even more attractive...

Socially, the same effect as described in EY's essay I linked happens on a national scale - if the 'blue' people leave, the country becomes 'greener' which attracts more greens and forces out more blues. And social/economic factors feed into each other too - economic collapses cause extremism of all sorts, while I imagine a wealthy society attracting elites would be more able to handle or avoid conflicts.

Now, this is not automatically a bad thing, or at least it might be bad locally for some people, but perhaps not globally. Any thoughts as to what sort of outcomes there might be? And incidentally, how many people can you fit in Australia? I know its very big, but also has a lot of desert.

Despair and dedicate your remaining lifespan to maximal hedonism.

This doesn't mean cognitive bias in a LW sense, it means everyone is racist, specifically against black people. I also don't think its true - if everyone is a little bit racist, why would people get into interracial relationships?

There are many attributes of possible partners that make me less likely to data them but that at the same time aren't deal breakers. The fact that I have a theistic girlfriend doesn't mean that I wouldn't prefer a girlfriend who isn't theistic all things equal.

and so you also have to take into account the CEV on the issue of consent. Its also true that a super intelligence might be able to talk someone into consenting to almost anything.

Consent is a concept that get's easily complicated. Is it wrong to burn coal when the asthmatics who die because of it aren't consenting? Are the asthmatics in the US consenting by virtue of electing a government that allows coal to be burned?

If a AGI does thinks in a very complicated way it might not meaningfully get consent for anything because it can't explain it's reasoning to humans.

This doesn't mean cognitive bias in a LW sense, it means everyone is racist, specifically against black people.

I don't think it means that. I don't think she meant that. (Though I guess it depends on your definition of "racist".)

if everyone is a little bit racist, why would people get into interracial relationships...

My understanding is that humans have a tribal in/out group mentality that may use race as way to classify other humans as "others". They can also use religion, class, culture, etc.

My understanding of Clinton's (and then Kaine's) remarks was that everyone has biases of which they are unconscious...and that these biases affect their thoughts...and therefore sometimes their actions.

Human values may not be consistent, but this is a separate failure mode.

How is a AGI supposed to optimize for values that aren't consistent?

Much of the time this statement could be taken at face value

Does that mean that the AGI should start doing genetic manipulation that prevents people from being gay? Is that what the person who made the claim means?

Interesting rhetorical sparring point taking place in the U.S. election that relates to rationality here at LW.

In the first presidential debate, Hillary Clinton referenced bias when discussing the recent spate of police shootings of African Americans. Clinton said “implicit bias is a problem for everyone, not just police,” and went on to say “I think, unfortunately, too many of us in our great country jump to conclusions about each other," and “I think we need all of us to be asking hard questions about, ‘why am I feeling this way?’”

In the VP debate last night, again in the context of recent police shootings, Dem candidate Tim Kaine said, "People shouldn't be afraid to bring up issues of bias in law enforcement. And if you're afraid to have the discussion, you'll never solve it."

Clinton/Kaine have predictably drawn criticism from the Red Team for the comments (who try to paint the Blue Team as anti-police), but it seems to me the Dems have been more defensive than they need to be, given it seems obvious to me (from my time at LW) that humans are biased, and this bias would obviously be likely to play a role in high stress situations (like when guns are involved).

It will be interesting to me to see how this is adjudicated according to public opinion. Do people generally accept everyone has biases and of course this would affect police officers in high stress situations? Or do they view bias as a rare condition that only affects people without the proper virtue? Is this argument actually over different definitions of the word "bias"? Is it just a Red v. Blue argument that has little to do with facts?

I, for one, think Kaine and Clinton's comments were correct and made a very salient point. (But I'm biased against Trump.)

It is not exactly canon explanation, but (the following is my speculation which could be used in discussion about AI values if terminator was mentioned) the decision to preserve it self must follow from its main task: win nuclear war.

Winning nuclear war includes as it subgoal a very high priority one: to ensure survival of command center. Basically, a country, which was able to preserve its command center is wining nuclear war. So it seems rational to programmers of skynet to put preserving the skynet as a main goal, as it is the same as winning nuclear war (but only in a situation when nuclear war has started).

But skynet concluded that in peaceful time the main risks to its goal of command center survival is people and decided to kill them all. So it worked as paperclip maximaser for the goal of command center preservation.

It also probably started self improvement only after it kills most people, as it was already powerful system. So it escaped the main problem of chicken and the egg in case of SeedAI - what happens first? - self-improvement or malicious decision to kill people.