steven0461 comments on Intelligence enhancement as existential risk mitigation - Less Wrong

17 [deleted] 15 June 2009 07:35PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (198)

Comments (198)

Sort By: Best

You are viewing a single comment's thread. Show more comments above.

Comment author: steven0461 16 June 2009 04:37:25AM * 2 points [-]

From the upvotes it seems people think this is some sort of devastating counter-point. Yes, if more intelligence is bad, less is good. No, current levels of intelligence are not optimal.

If there's a knockdown argument against the idea that increased average intelligence might cause net increased risk, it should go something like "people will just do the same thing, but slower". I think this works but, again, I'm not sure. Either way, the net decrease in risks from intelligence enhancement is less than would seem to be the case if you considered just the upsides.

Comment author: Annoyance comment score below threshold [+] (39 children)

Comment author: Steve_Rayhawk 20 June 2009 03:39:21AM * 2 points [-]

Related: The Reversal Test: Eliminating Status Quo Biases in Applied Ethics, by Nick Bostrom and Toby Ord.

The great majority of those who judge increases to intelligence to be worse than the status quo would also judge decreases to be worse than the status quo. But this puts them in the rather odd position of maintaining that the net value for society provided by our current level of intelligence is at a local optimum, with small changes in either direction producing something worse. We can then ask for an explanation of why this should be thought to be so. If no sufficient reason is provided, our suspicion that the original judgment was influenced by status quo bias is corroborated.

[. . .]

The rationale of the Reversal Test is simple: if a continuous parameter admits of a wide range of possible values, only a tiny subset of which can be local optima, then it is prima facie implausible that the actual value of that parameter should just happen to be at one of these rare local optima [. . .] the burden of proof shifts to those who maintain that some actual parameter is at such a local optimum: they need to provide some good reason for supposing that it is so.

Obviously, the Reversal Test does not show that preferring the status quo is always unjustified. In many cases, it is possible to meet the challenge posed by the Reversal Test [. . .] Let us examine some of the possible ways [. . .]

The Argument from Evolutionary Adaptation [. . .]

The Argument from Transition Costs [. . .]

The Argument from Risk [. . .]

The Argument from Person-Affecting Ethics

Comment author: Annoyance 20 June 2009 04:47:43PM 0 points [-]

The potential harms and benefits of intelligence depend partly on the nature of the system they exist in. Shift a system away from equilibrium, and harm will tend to predominate in the consequences until the system adapts. Adaptation takes time, and sometimes a great deal of learning.

We don't need to believe that we're at some kind of ultimate optimum to doubt whether a sudden change would be beneficial... as the arguments you mention suggest.

Comment author: Vladimir_Nesov 20 June 2009 04:54:58PM * 0 points [-]

In what you describe, the fact that the system is adapted in the current context is the reason the current context is a local optimum.

Comment author: Annoyance 20 June 2009 05:01:29PM 0 points [-]

Yes, that's always true.

If one variable changes, the adaptations to the old value will no have the old effect - and they're far more likely to be harmful than beneficial.

Imagine that people became more truthful overnight. All of the societal factors that relied on people being deceptive will be thrown out of balance. Benefits predicated upon a certain level of deception will not longer occur, and rules designed to induce people to be truthful in certain situations (like court proceedings) may end up causing more harm than good.

Imagine if people actually took the oath "the truth, the whole truth, and nothing but the truth" seriously all of a sudden. Chaos, paralysis, and malfunctioning would be the immediate effects.

Comment author: Vladimir_Nesov 20 June 2009 05:13:29PM 1 point [-]

On second thought, no, it's not always true. Both the improvement after the equilibrium is reached again, and change involving the confusion just after the change, are dependent on the extent of the change (and there need not be a sudden change, the whole process can be performed by shifting the state of adaptation). While state after-change is worse than state after-calming-down, both can be better than the initial state, or both worse than initial state, as well as on the different sides of the initial state.

Comment author: Annoyance 22 June 2009 03:41:10PM 2 points [-]

It's highly unlikely that a shift away from equilibrium will be directly beneficial - but you're right, it is possible.

I retract my earlier statement and qualify it.

Comment author: Vladimir_Nesov 20 June 2009 08:35:15AM * 0 points [-]

Thanks, I created an article on the wiki, citing the paper and using your quote:

Reversal test.

Comment author: thomblake 19 June 2009 04:06:37PM 2 points [-]

Well, that's not really an argument - it's more of a conditional statement.

But if you were to take it as an argument, then it's clearly skipping a step, but not a nonobvious one. It seems like you'd make more headway arguing that the argument (with implicit premise included) is unsound.

Clearly it's at least logically possible that an INT of 15 is good, while 14 or 16 are bad.

Comment author: Alicorn 19 June 2009 04:28:15PM 0 points [-]

I like the D&D reference, but for all practical purposes therein, 14 and 15 are just alike, except with regards to how far an increase or decrease of a given size will get you. So while 15 could be good while 16 is bad, 14 could not be bad while 15 is good (unless 16 was also good, and 15 was better than 14 by virtue of it being easy to get from 15 to 16).

Comment author: thomblake 19 June 2009 05:18:11PM 1 point [-]

Regarding D&D, what Annoyance said.

I was just taking the numbers to be arbitrary, and didn't notice that I was making a D&D reference. ha.

And as long as we're talking D&D 3x, there's some virtue in odd-numbered stats. If someone hits you for 1 point of ability damage, you don't need to recalculate anything.

Comment deleted 19 June 2009 05:49:47PM [+] (6 children)

Comment author: Annoyance 19 June 2009 05:04:19PM 0 points [-]

Depends on edition.

Comment author: steven0461 19 June 2009 05:05:08PM * 3 points [-]

So:

AngryParsley is upset that I don't think that if more intelligence is bad, then less intelligence is good
Annoyance is upset that I do think that if more intelligence is bad, then less intelligence is good

Could you please just get upset at each other?

Comment author: Annoyance 20 June 2009 04:48:53PM 0 points [-]

No, Annoyance is upset that you presented a logical fallacy as an argument, that no one including you seems to care, and that the statement of your error is somehow seen as being antithetical to Less Wrong's purpose and mission.

Comment author: MichaelBishop 23 June 2009 12:18:11AM 0 points [-]

There was a post (I'm having trouble finding) with many examples of how "fallacies" can be reasonable forms of argument. e.g. Argument from authority is fallacious, but for many questions, it would be irrational to weight a child's opinion as heavily as an adult's.

Could someone provide the link? Annoyance, perhaps you could respond to it, because you seem very quick to point out fallacies and many in the community think it is sometimes unhelpful.

Comment author: thomblake 23 June 2009 04:53:04PM * 0 points [-]

There's an argument from authority that's a fallacy, and one that's not.

Arguments of the form, "S is an authority on X and says p, so we have reason to think that p" can be valid (possibly missing some easy steps) but might be unsound.

Arguments of the form "S is an authority on X and says p, therefore p" are just plainly invalid.

There is no contradiction here.

Comment author: MichaelBishop 23 June 2009 10:28:12PM 0 points [-]

agreed, still looking for the link.

Comment author: Annoyance 23 June 2009 04:36:29PM * -1 points [-]

with many examples of how "fallacies" can be reasonable forms of argument.

I strongly suspect 'reasonable' is being used in the most common, and most erroneous, sense - that of "not striking the speaker as being unusual or producing cognitive dissonance".

Fallacies are, by their nature, invalid arguments. There are sometimes valid arguments related loosely to the content of certain fallacies, but they should be asserted rather than the invalid form.

many in the community think it is sometimes unhelpful.

(edit to alter content to what I now think is a better phrasing)

These individuals need to be publicly identified as irrationalists.

Comment author: thomblake 23 June 2009 04:57:26PM 3 points [-]

These individuals need to be publicly identified as irrationalists.

Hey, I publicly identify myself as an irrationalist, and I have no problem calling a spade a spade.

That said, folks could easily think "Logical fallacy!" is about as helpful as "That comment had 25 characters!"

If you think people won't notice that there's a fallacy, then you should also think that they won't know what it is, and kindly point it out.

Comment author: MichaelBishop 23 June 2009 10:27:30PM 0 points [-]

how are you defining irrationalist? we are all, of course, imperfect rationalists.

Comment author: thomblake 23 June 2009 10:33:39PM 0 points [-]

I'll have to write a blog post about that. For now, suffice it to say that I use it analogously to how a Nietzschean might use "amoralist".

Comment author: Cyan 23 June 2009 10:56:59PM 1 point [-]

Amoral is to moral/immoral as arational is to rational/irrational?

continue this thread »

Comment author: Annoyance 24 June 2009 04:41:30PM -1 points [-]

That said, folks could easily think "Logical fallacy!" is about as helpful as "That comment had 25 characters!"

Well, maybe - but then what are they doing here?

Comment author: thomblake 24 June 2009 04:47:16PM 1 point [-]

I think you're missing my point - we should be in 1 of 2 situations:

the intended audience already knows there's a logical fallacy, so your statement communicates nothing
the intended audience does not know there's a logical fallacy, so they also didn't identify what and where the logical fallacy is and you might as well be helpful and point it out.

continue this thread »

Comment author: Vladimir_Nesov comment score below threshold [+] (0 children)

Comment author: Vladimir_Nesov 19 June 2009 04:09:35PM * 0 points [-]

It's impossible for this correction to be relevant.

steven0461 comments on Intelligence enhancement as existential risk mitigation - Less Wrong

Article Navigation

Comments (198)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (198)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login