steven0461 comments on Intelligence enhancement as existential risk mitigation

Comment author: AngryParsley 15 June 2009 10:48:46PM 7 points [-]

If more intelligence is bad, is less good? Do you think current levels of intelligence are optimal? If so, that would be an amazing coincidence.

Comment author: Peter_de_Blanc 16 June 2009 04:57:38AM 3 points [-]

I don't think that current levels of intelligence are optimal, but if they were, it wouldn't be a coincidence. Humans are adaptation-executers, and genes make implicit assumptions about their environment. In particular, certain adaptations might be disrupted by changing the average intelligence.

Comment author: AngryParsley 16 June 2009 09:49:38AM * 3 points [-]

If you had the option to increase your intelligence, would you decline because you were worried about certain adaptations being disrupted? The modern world is so different from our EEA that I can't buy your argument.

Disrupting adaptations can be a good thing. Birth control helps prevent overpopulation. Courts help settle disputes without violence. Even rational thought involves recognizing and changing (disrupting?) the typical thought patterns of our adapted brains.

Comment author: Vladimir_Nesov 16 June 2009 12:58:28PM * 4 points [-]

The fact that modern world changed our values in a way that ancient people won't appreciate on reflection is a bad thing for the ancient people. To us, it'd be bad if we reverted some of these changes, and likewise if we introduced new changes that have negative side effects from the current point of view (on reflection).

It's hard to "increase intelligence" without wreaking some of the values, brain isn't designed for upgrade. It's the same problem as with trying to change emotions.

Comment deleted 16 June 2009 01:15:25PM [-]

Comment author: MichaelBishop 17 June 2009 12:52:22AM 2 points [-]

I definitely think that values, however defined, would change significantly with such a 10 point IQ change (btw, I consider this very large). And I think it would probably be a good thing.

Comment author: steven0461 16 June 2009 04:37:25AM * 2 points [-]

From the upvotes it seems people think this is some sort of devastating counter-point. Yes, if more intelligence is bad, less is good. No, current levels of intelligence are not optimal.

If there's a knockdown argument against the idea that increased average intelligence might cause net increased risk, it should go something like "people will just do the same thing, but slower". I think this works but, again, I'm not sure. Either way, the net decrease in risks from intelligence enhancement is less than would seem to be the case if you considered just the upsides.

Comment author: Annoyance comment score below threshold [+] (39 children)

Comment author: Steve_Rayhawk 20 June 2009 03:39:21AM * 2 points [-]

Related: The Reversal Test: Eliminating Status Quo Biases in Applied Ethics, by Nick Bostrom and Toby Ord.

The great majority of those who judge increases to intelligence to be worse than the status quo would also judge decreases to be worse than the status quo. But this puts them in the rather odd position of maintaining that the net value for society provided by our current level of intelligence is at a local optimum, with small changes in either direction producing something worse. We can then ask for an explanation of why this should be thought to be so. If no sufficient reason is provided, our suspicion that the original judgment was influenced by status quo bias is corroborated.

[. . .]

The rationale of the Reversal Test is simple: if a continuous parameter admits of a wide range of possible values, only a tiny subset of which can be local optima, then it is prima facie implausible that the actual value of that parameter should just happen to be at one of these rare local optima [. . .] the burden of proof shifts to those who maintain that some actual parameter is at such a local optimum: they need to provide some good reason for supposing that it is so.

Obviously, the Reversal Test does not show that preferring the status quo is always unjustified. In many cases, it is possible to meet the challenge posed by the Reversal Test [. . .] Let us examine some of the possible ways [. . .]

The Argument from Evolutionary Adaptation [. . .]

The Argument from Transition Costs [. . .]

The Argument from Risk [. . .]

The Argument from Person-Affecting Ethics

Comment author: Annoyance 20 June 2009 04:47:43PM 0 points [-]

The potential harms and benefits of intelligence depend partly on the nature of the system they exist in. Shift a system away from equilibrium, and harm will tend to predominate in the consequences until the system adapts. Adaptation takes time, and sometimes a great deal of learning.

We don't need to believe that we're at some kind of ultimate optimum to doubt whether a sudden change would be beneficial... as the arguments you mention suggest.

Comment author: Vladimir_Nesov 20 June 2009 04:54:58PM * 0 points [-]

In what you describe, the fact that the system is adapted in the current context is the reason the current context is a local optimum.

Comment author: Annoyance 20 June 2009 05:01:29PM 0 points [-]

Yes, that's always true.

If one variable changes, the adaptations to the old value will no have the old effect - and they're far more likely to be harmful than beneficial.

Imagine that people became more truthful overnight. All of the societal factors that relied on people being deceptive will be thrown out of balance. Benefits predicated upon a certain level of deception will not longer occur, and rules designed to induce people to be truthful in certain situations (like court proceedings) may end up causing more harm than good.

Imagine if people actually took the oath "the truth, the whole truth, and nothing but the truth" seriously all of a sudden. Chaos, paralysis, and malfunctioning would be the immediate effects.

Comment author: Vladimir_Nesov 20 June 2009 05:13:29PM 1 point [-]

On second thought, no, it's not always true. Both the improvement after the equilibrium is reached again, and change involving the confusion just after the change, are dependent on the extent of the change (and there need not be a sudden change, the whole process can be performed by shifting the state of adaptation). While state after-change is worse than state after-calming-down, both can be better than the initial state, or both worse than initial state, as well as on the different sides of the initial state.

Comment author: Annoyance 22 June 2009 03:41:10PM 2 points [-]

It's highly unlikely that a shift away from equilibrium will be directly beneficial - but you're right, it is possible.

I retract my earlier statement and qualify it.

Comment author: Vladimir_Nesov 20 June 2009 08:35:15AM * 0 points [-]

Thanks, I created an article on the wiki, citing the paper and using your quote:

Reversal test.

Comment author: thomblake 19 June 2009 04:06:37PM 2 points [-]

Well, that's not really an argument - it's more of a conditional statement.

But if you were to take it as an argument, then it's clearly skipping a step, but not a nonobvious one. It seems like you'd make more headway arguing that the argument (with implicit premise included) is unsound.

Clearly it's at least logically possible that an INT of 15 is good, while 14 or 16 are bad.

Comment author: Alicorn 19 June 2009 04:28:15PM 0 points [-]

I like the D&D reference, but for all practical purposes therein, 14 and 15 are just alike, except with regards to how far an increase or decrease of a given size will get you. So while 15 could be good while 16 is bad, 14 could not be bad while 15 is good (unless 16 was also good, and 15 was better than 14 by virtue of it being easy to get from 15 to 16).

Comment author: thomblake 19 June 2009 05:18:11PM 1 point [-]

Regarding D&D, what Annoyance said.

I was just taking the numbers to be arbitrary, and didn't notice that I was making a D&D reference. ha.

And as long as we're talking D&D 3x, there's some virtue in odd-numbered stats. If someone hits you for 1 point of ability damage, you don't need to recalculate anything.

Comment deleted 19 June 2009 05:49:47PM [+] (6 children)

Comment author: Annoyance 19 June 2009 05:04:19PM 0 points [-]

Depends on edition.

Comment author: steven0461 19 June 2009 05:05:08PM * 3 points [-]

So:

AngryParsley is upset that I don't think that if more intelligence is bad, then less intelligence is good
Annoyance is upset that I do think that if more intelligence is bad, then less intelligence is good

Could you please just get upset at each other?

Comment author: Annoyance 20 June 2009 04:48:53PM 0 points [-]

No, Annoyance is upset that you presented a logical fallacy as an argument, that no one including you seems to care, and that the statement of your error is somehow seen as being antithetical to Less Wrong's purpose and mission.

Comment author: MichaelBishop 23 June 2009 12:18:11AM 0 points [-]

There was a post (I'm having trouble finding) with many examples of how "fallacies" can be reasonable forms of argument. e.g. Argument from authority is fallacious, but for many questions, it would be irrational to weight a child's opinion as heavily as an adult's.

Could someone provide the link? Annoyance, perhaps you could respond to it, because you seem very quick to point out fallacies and many in the community think it is sometimes unhelpful.

Comment author: thomblake 23 June 2009 04:53:04PM * 0 points [-]

There's an argument from authority that's a fallacy, and one that's not.

Arguments of the form, "S is an authority on X and says p, so we have reason to think that p" can be valid (possibly missing some easy steps) but might be unsound.

Arguments of the form "S is an authority on X and says p, therefore p" are just plainly invalid.

There is no contradiction here.

Comment author: MichaelBishop 23 June 2009 10:28:12PM 0 points [-]

agreed, still looking for the link.

Comment author: Annoyance 23 June 2009 04:36:29PM * -1 points [-]

with many examples of how "fallacies" can be reasonable forms of argument.

I strongly suspect 'reasonable' is being used in the most common, and most erroneous, sense - that of "not striking the speaker as being unusual or producing cognitive dissonance".

Fallacies are, by their nature, invalid arguments. There are sometimes valid arguments related loosely to the content of certain fallacies, but they should be asserted rather than the invalid form.

many in the community think it is sometimes unhelpful.

(edit to alter content to what I now think is a better phrasing)

These individuals need to be publicly identified as irrationalists.

Comment author: thomblake 23 June 2009 04:57:26PM 3 points [-]

These individuals need to be publicly identified as irrationalists.

Hey, I publicly identify myself as an irrationalist, and I have no problem calling a spade a spade.

That said, folks could easily think "Logical fallacy!" is about as helpful as "That comment had 25 characters!"

If you think people won't notice that there's a fallacy, then you should also think that they won't know what it is, and kindly point it out.

Comment author: MichaelBishop 23 June 2009 10:27:30PM 0 points [-]

how are you defining irrationalist? we are all, of course, imperfect rationalists.

continue this thread »

Comment author: Vladimir_Nesov comment score below threshold [+] (0 children)

Comment author: Vladimir_Nesov 19 June 2009 04:09:35PM * 0 points [-]

It's impossible for this correction to be relevant.

Comment author: Arenamontanus 16 June 2009 06:12:06PM 2 points [-]

More intelligence means bigger scope for action, and more ability to get desired outcomes. Whether more intelligence increases risk depends on the distribution of accidentally bad outcomes in the new scope (and how many old bad outcomes can be avoided), and whether people will do malign things. On average very few people seem to be malign, so the main issue is likely more the issue of new risks.

Looking at the great deliberate human-made disasters of the past suggests that they were often more of a systemic nature (societies allowing nasty people or social processes to run their course; e.g. democides and genocides) than due to individuals or groups successfully breaking rules (e.g. terrorism). This is actually a reason to support cognitive enhancement if it can produce more resilient societies less prone to systemic risks.

Comment author: steven0461 16 June 2009 08:18:58PM * 0 points [-]

One possibility I have in mind is if current rationalist ideas need a certain amount of time to slosh around and pervade the population before technology (fed by intelligence) grows enough for them to really start mattering.

Comment author: gwern 17 June 2009 03:39:14AM 1 point [-]

It does seem to me like the anti-risk effect from intelligence enhancement would somewhat outweigh the pro-risk effect, but I'm not sure.

It is a hard subject to argue about. If I wanted to criticize you, I could say that higher IQs make uploads less economically attractive as they will start off with a smaller advantage, and higher IQs likely would make uploads intrinsically more difficult. And if I wanted to criticize my criticism, I could say that by making people in general smarter, we increase the cost of labor, which makes mechanical substitution ever more attractive (and mechanical substitution for reasonably complicated tasks implies development of AI/uploads).

Comment author: HughRistik 15 June 2009 11:48:10PM 0 points [-]

Yes, increasing intelligence would increase the variance of quality of outcomes for humanity. And the hope is that intelligence would also increase the mean quality of outcome, such that the expected value would be higher.

steven0461 comments on Intelligence enhancement as existential risk mitigation - Less Wrong

Article Navigation

Comments (198)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (198)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login