Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: lukeprog 19 August 2014 05:10:19AM 13 points [-]

MIRI intends to make an AI that is provably friendly.

I really wish people would stop repeating this claim. Mathematical Proofs Improve But Don’t Guarantee Security, Safety, and Friendliness.

Comment author: V_V 20 August 2014 08:12:20AM *  1 point [-]

And yet, all the publicly known MIRI research seems to be devoted to formal proof systems, not to testing, "boxing", fail-safe mechanisms, defense in depth, probabilistic failure analysis, and so on.

Motte and bailey?

Comment author: ShardPhoenix 19 August 2014 08:44:17AM *  8 points [-]


Prevalence: Between 13 - 28%. My guess is about 13%.

Wow. That's at least an order of magnitude higher than I would have guessed. I even thought you might have accidentally omitted a decimal point at first.

Comment author: V_V 19 August 2014 08:34:29PM *  6 points [-]

I had the same thought.

Indeed the prevalence in the general US population is 50x smaller: http://www.cdc.gov/hiv/statistics/basics/
I knew that MSM were a high risk group, but I didn't realize the risk was that high.

Comment author: RichardKennaway 19 August 2014 08:48:01AM 3 points [-]

So one could dismiss the whole thing as a bad version of Christianity or some other religion.

Only by ignoring the fundamental question: Is it true?

Comment author: V_V 19 August 2014 05:24:09PM *  1 point [-]

The point is that all object-level arguments for and against these scenarios, even if you call them "probability estimates", are ultimately based on intuitions which are difficult to formalize or quantify.

The scenarios hypthesized by the Singularitarians are extreme, both in the magnitude of the effect they are claimed to entail, and in the the highly conjunctive object-level arguments that are used to argue for them. Common sense rationality tells us that "extraordinary claims demand exceptional evidence". How do we evaluate whether the intuitions of these people constitute "exceptional evidence"?

So we take the "outside view" and try to meta-reason on these arguments and the people making them:
Can we trust their informal intuitions, or do they show any signs of classical biases?

Are these people privileging the hypothesis? Are they drawing their intuitions from the availability heuristic?

If intelligence explosion/cryonics/all things singularitarian were ideas radically different from any common meme, then the answer to these questions would be likely no: these ideas would appear counterintuitive at a gut level to most normally rational people, possibly in the same way quantum mechanics and Einstenian relativity appear conterituitive.
If domain-level experts, after studing the field for years, recalibrated their intuitions and claimed that these scenarios were likely, then we should probably listen to them.
We should not just accept their claims based on authority, of course: even the experts can subject to groupthink and other biases (cough...economists...cough), but as far as the "outside view" is concerned, we would at least have plausibly excluded the availability bias.

What we observe, instead, is that singulariarians ideas strongly pattern-match to Christian millenarianism and similar religious beliefs, mixed with popular scifi tropes (cryonics, AI revolt, etc.). They certainly originated, or at least were strongly influenced by these memes, and therefore the intuitions of the people arguing for them are likely "contaminated" via the availability heuristic by these memes.
More specifically, if singulariarians ideas make intuitive sense to you, you can't even trust your own intuitions since they are likely to be "contaminated" as well.

Add the fact that the strength of these intuitions seems to decrease rather than increase with domain-expertise, suggesting that the Dunning–Kruger effect is also at work, then the "outside view" tells us to be wary.

Of course, it is possible to believe correct things even when they are likely to be the subject of biases, or even to believe correct things that many people believe for the wrong reason, but in order to make a case for these beliefs, you need some airtight arguments with strong evidence.
As far as I can tell, MIRI/FHI/other Singularitarians have provided no such arguments.

Comment author: gwern 17 August 2014 10:27:34PM 1 point [-]

Singapore is a small country which deliberately attracts elites and tries to practice eugenics; so I don't think that's a very good example at all to use against a statistical generalization...

Comment author: V_V 18 August 2014 09:33:38AM 0 points [-]


But China is also pretty smart, and as far as I know it doesn't have a North-South IQ gradient: http://akarlin.com/2012/08/analysis-of-chinas-pisa-2009-results/

According to Wikipedia, the genetics of the Han Chinese is... complicated.
But even if high-IQ genes were ancestral in northen Hans and then were transferred to southern Hans due to migrations, if warm climates selects negatively for intelligence I think we should expect that in the last 2,000 years those high-IQ genes would not have thrived in South China.

Comment author: Vaniver 15 August 2014 08:11:34PM *  3 points [-]

I don't have any hard data,

Did you look for any?

Rushton and Ankney (2009) summarize the findings to date with regards to brain size and intelligence: based on 28 non-clinical published brain imaging samples (N= 1,389) a .40 correlation between IQ and brain size measured by MRI was found; based on 59 published samples (N= 63,405) a .20 correlation between IQ and head circumference was found. These findings are consistent with others.

Quoted from here, the paper is here (they should have quoted the correlation of 0.38, which is what you get when you weight by sample size).

It's obvious that mental tasks do consume glucose. Jensen mentions metabolic correlations here, but not which direction they go in. This paper suggests that IQ and cerebral glucose metabolic rate are inversely correlated, and that after learning a new task more intelligent individuals showed larger decreases, but it looks like it has a very low n and I'd want to draw conclusions from review papers rather than individual investigations. I would not be surprised if the brain efficiency hypothesis dominates, and that higher IQ individuals get more bang for the buck instead of burning more to get more. I also hear more about cooling costs than calorie costs with regards to brain metabolism, but that may be because cooling costs fits with the observed data of smarter people evolving in colder places with higher latitudes.

Comment author: V_V 17 August 2014 08:18:30AM 1 point [-]

the observed data of smarter people evolving in colder places with higher latitudes.

Like Singapore?

Comment author: Azathoth123 16 August 2014 04:02:45AM 1 point [-]

I've been thinking about the issue in terms of (armchair) game theory:

It would help to actually look at the history rather than simply completing the pattern and pretending to be wise.

Ostensibly, both parties essentially claim to be playing a a tit-for-tat strategy in an interated prisoner dilemma where the other party is a DefectBot that can't be cooperated with.

Israel has an various periods starting with Yitzhak Rabin attempted to play unilateral cooperation under the theory that this would lead to cooperation on the other end. Hasn't worked out that way.

Comment author: V_V 16 August 2014 07:15:45AM *  0 points [-]

It would help to actually look at the history rather than simply completing the pattern and pretending to be wise.

"This s our land, and anybody who suggests that somebody else might have some modicum of claim over it must be pretending to be wise and <insert teacher's password here>."

Israel has an various periods starting with Yitzhak Rabin attempted to play unilateral cooperation under the theory that this would lead to cooperation on the other end. Hasn't worked out that way.

"Anything we might concede you is more than you deserve and you'd better be grateful and accept it without question before we change our mind."

Comment author: eli_sennesh 14 August 2014 06:48:46AM 3 points [-]

As often happens, the truth is more complex than either of you are giving credit for.

Neither Fatah nor Hamas has much real popular support, at this point. I agree that Fatah has in fact been making credible efforts towards negotiating a peace deal, actually (and I say this as an Israeli). Hamas, yes, they are in fact genocidally-inclined clerical fascists, Sharia law blah blah, executing dissenters in the streets, blah blah.

But Hamas has one thing going for them: they fight the Israelis. Fatah does not, certainly not as much as the Palestinian public would prefer. So the current situation is: most Palestinians favor the PLO's traditional ideology of semi-secular nationalism, and believe in the PLO's traditional historiography, but consider the Fatah organization itself to have become corrupt stooges for the Israeli occupier. Some of them channel this belief into despair, some into political anarchism, and many into a quiet, tacit support for Hamas militancy.

Netanyahu's nonstop bad-faith "negotiation" ever since his election and reelection has not been helping.

Comment author: V_V 15 August 2014 01:29:43PM *  1 point [-]

I've been thinking about the issue in terms of (armchair) game theory:

Ostensibly, both parties essentially claim to be playing a a tit-for-tat strategy in an interated prisoner dilemma where the other party is a DefectBot that can't be cooperated with.
Who played "Defect" first and when is a matter of dispute: arguments usually involve to events going back to at least the British Empire, if not the Ottoman Empire or even the Roman Empire.
Regardless of who started it, both parties think (or at least claim to think) that they can't break the defection cycle by attempting to cooperate unilaterally, hence they defect.

Obviously this analysis is simplistic since it models the Israeli and the Palestinians each as a single agent. But since, as you say, Palestinians tend to more or less unwillingly support Hamas and the Israeli tend to support Netanyahu and his coallition government (also more or less unwillingly?), I think this two-agent model is a reasonable first-order approximation.

Another issue is that there doesn't appear to be a Schelling point that both parties recognize as a "default" solution for the bargaining problem they face. It seems that both parties operate under a framework of "This land is ours. You have no right to be here. Anything we might concede you is more than you deserve and you'd better be grateful and accept it without question before we change our mind."
This isn't conducive to productive negotiation.

Comment author: Jiro 12 August 2014 02:26:54PM *  -2 points [-]

There is a reason why getting rid of a murderer should not cost far more lives than projected deaths from the murderer.

There's a reason why police and military are not the same thing. I don't remember offhand how many people died on the Pacific front in World War II, but I'm pretty sure it's more than died at Pearl Harbor. And at least at Pearl Harbor the US had the option of not entering the war and not getting any more Americans killed. The idea that you should not kill any more of the enemy than the enemy kills of you is something you and a lot of other people have basically made up. (And why aren't you counting the projected deaths from the murderer as the murderer killing a few people at a time until forever, thus resulting in infinite projected deaths anyway?)

Also, what do you do if the enemy decides to deliberately surround himself with civilians so as to maximize the civilian casualties from attacking him?

Comment author: V_V 12 August 2014 04:37:02PM *  -2 points [-]

The Imperial Japanese Army posed a significant threat to US interests and long term security, expecially given that it was allied to the other Axis powers. Still, many people believe that the nuking of Hiroshima and Nagasaki was excessive and immoral.

Anyway, WWII is a poor analogy for the Israel-Gaza conflict: the threat level Hamas exerts on Israeli security, or even economic welfare, is minimal, therefore the damages to the security ang welfare of Gazans caused by Israel "self-defence" operations are seen as greatly disproportionate.

In fact, it is often argued that Israel military actions are even inefficient at accomplishing their stated goal of protecting the security of its citizens, as opposed to other non-military options that Israel has, such as avoiding to provoke Hamas to fire rockets by imprisoning and killing its members in the West Bank without good cause, or finding reasonable terms for a truce (face-saving for both parties) once the mess had been started.
In the medium-long term, it can be argued that it would be in the interests of the security of Israeli citizen that their government negotiated a permanent political solution with both Fatah and Hamas.

Another important way in which analogies based on traditional warfare fail, is that Gaza or the West Bank aren't countries separate from Israel, they are regions under Israel military occupation.
Gaza has some level of autonomy, with Hamas acting as a de facto government for day-to-day administration and law enforcement, but Israel keeps control over Gaza airspace and access to international waters, which still counts as military occupation under international law. International law, which many people find morally reasonable, also says that an occuping powers has special duties to protect the civilians of the regions under occupations: since they don't have a fully sovereign government taking care of their interests, the occupying power must take the burden on itself.

In any case, international law, and I would say common morality as well, condemns the killing of civilians if it doesn't accomplish a reasonable and "proportionate" military objective.
The Allied firebombing of Dresden, and maybe even the nuking of Hiroshima and Nagasaki, can be argued to have accomplished valuable and proportionate military objectives despite the high number of civilian deaths, as they greatly damaged the enemy industrial capability and hence their war effort.
In the current round of Israel-Gaza conflict, thousands of Gazan civilians have been killed, including children playing football on a beach, people in the premises of an UN school, etc. Israel officially claims that some of these deaths were accidents and blames the others on Hamas usage of human shields. Maybe that's techically true, but when you are bombing and shelling an heavily populated area, a certain amount of "accidents" and more or less deliberate human shielding are inevitable. You know that a lots of civilians are going to die, and you need a damm good military objective in order to justify it.
What military objective is Israel accomplishing? Hamas is still firing its rockets, which are still extremely ineffective at killing Israeli citizens. Eventually, Israel may stop the Hamas rocket launches by destroying all the caches and smuggling tunnels. And it will take only a couple of years for Hamas to replenish its arsenal, and for each dead militant, Hamas will be able to recruit three from the enraged population.

This combination of large loss of civilian lives and minimal accomplishment of military objectives, particularly given the status of Gaza as an occupied territory, are the reason why people argue that Israel actions are disproportionate and doubt Israel good faith in its claims that it is trying to minimize civilian losses.

The alternate, uncharitable hypothesis to explain Israel behavior is that it is trying to maximize Gazan civilian deaths while maintaining the plausible deniability it needs in order to keep US financial and military aid and support in the UN Security Council. Thus Israel can't nuke the Gazans or put them into death camps, but, according to this model, it would love to.

This model, where all the Israeli are bloodthirsty monsters, is overly uncharitable. Real world politics is complicated:
Israel has an almost pure proportional political representation system, which results in coallition governments where small parties that can switch sides are greatly favoured. The current ruling coallitions has nationalistic and religious parties which are apparently made of kill-all-Arabs bloodthirsty monsters, and the government as a whole has to balance padering to a broad electorate, which includes bloodthirsty monsters that vote for said parties.
Overall, the average Israeli voter is probably not a monster, but finds it easier to cooperate with the Israeli monsters rather than with the Palestinians, who also have their share of kill-all-Jews monsters.

Anyway, this comment is quite long and discussion of contemporary object-level real-world politics is discouraged here, probably with good reason. The Israeli-Gaza conflict is one of the most divisive political topic ever, therefore I'm not going to carry on this discussion any more.

Comment author: Jiro 11 August 2014 08:06:24PM 0 points [-]

This is not inconsistent with the claim that, if the country of rationalists is attacked by a country of barbarians who are imperfectly optimized for conquest, the rationalists might get conquered, with the risk depending on how optimized the barbarians are.

How imperfectly optimized, though? Imperfectly optimized like Omega controlling each barbarian but occasionally rolling the barbarian's morale check, which fails on a 1 on a D100? Or imperfectly optimized like real life barbarians?

Comment author: V_V 11 August 2014 08:44:40PM 1 point [-]

What about the Bolsheviks? Or the WW2-era Japanese?

Comment author: Jiro 11 August 2014 07:39:04PM -3 points [-]

How many Israeli civilians have been killed by rockets since Operation Protective Edge? Three. Over a population of 8.1 million.

How many people did John Hinckley, Jr. kill? One. Out of a population of, oh, 300 million.

Comment author: V_V 11 August 2014 07:54:52PM *  -1 points [-]

How many people did John Hinckley, Jr. kill? One.

According to Wikipedia, he killed none and injuried four, including the President of the United States. How many people did the US government kill in order to "self-defend" from John Hinckley, Jr?

View more: Next