Psy-Kosh comments on The Sword of Good - Less Wrong

85 Post author: Eliezer_Yudkowsky 03 September 2009 12:53AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (292)

Comments (292)

Sort By: Best

Show:

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 03 September 2009 07:22:11PM 12 points [-]

go for it themselves

What, you mean try to self-modify? Oh hell no. Human brain not designed for that. But you would have a longer time to try to solve FAI. You could maybe try a few non-self-modifications if you could find volunteers, but uploading and upload-driven-upgrading is fundamentally a race between how smart you get and how insane you get.

Comment author: Psy-Kosh 03 September 2009 07:35:40PM * 3 points [-]

*blinks* I understand your "oh hell no" reaction to self modification and "use the speedup to buy extra time to solve FAI" suggestion.

However, I don't quite understand why you think "attempted upgrading of other" is all that much better. If you get that one wrong in a "result is super smart but insane (or, more precisely, very sane but with the goal architecture all screwed up) doesn't one end up with the same potential paths to disaster? At that point, if nothing else, what would stop the target from then going down the self modification path?

Comment author: Eliezer_Yudkowsky 03 September 2009 11:02:01PM 7 points [-]

Non-self-modification is by no means safe, but it's slightly less insanely dangerous than self-modification.

Comment author: pjeby comment score below threshold [+] (23 children)

Comment author: Eliezer_Yudkowsky 04 September 2009 04:58:23AM 7 points [-]

We're talking about direct brain editing here. People who insist on comparing direct brain editing to various forms of internal rewiring carried out autonomously by opaque algorithms... or choice over deliberate procedures to follow deliberatively... well, don't be surprised if you're downvoted, because you did, in fact, say something stupid.

Comment author: pjeby 05 September 2009 01:03:56AM -1 points [-]

We're talking about direct brain editing here.

If by "direct" here you mean changing the underlying system - metaprogramming as it were, then I have to say that that's the idea that's stupid. If you have a system that's perfectly capable of making changes on its own, debugged by millions of years of evolution, why on earth would you want to bypass those safeties?

On that, I believe we're actually in agreement.

Comment author: Vladimir_Nesov 05 September 2009 08:11:45AM 6 points [-]

If you have a system that's perfectly capable of making changes on its own, debugged by millions of years of evolution, why on earth would you want to bypass those safeties?

To do better?

Comment author: pjeby 05 September 2009 01:30:54PM 1 point [-]

To do better?

You don't need to bypass the safeties to do better. What you need is not a bigger hammer with which to change the brain, but a better idea of what to change, and what to change it to.

That's the thing that annoys me the most about brain-mod discussions here -- it's like talking about opening up the case on your computer with a screwdriver, when you've never even looked at the screen or tried typing anything in -- and then arguing that all modifications to computers are therefore difficult and dangerous.

Comment author: CronoDAS 05 September 2009 10:43:47PM 2 points [-]

To use an analogy, the kind of brain modifications we're talking about would be the kind of modifications you'd have to do to a 286 in order to play Crysis (a very high-end game) on it.

Comment author: [deleted] 06 September 2009 02:01:56AM 1 point [-]

If I'm not mistaken, as far as raw computing power goes, the human brain is more powerful than a 286. The question is--and this is something I'm honestly wondering--whether it's feasible, given today's technology, to turn the brain into something that can actually use that power in a fashion that isn't horribly indirect. Every brain is powerful enough to play dual 35-back perfectly (if I had access to brain-making tools, I imagine I could make a dual 35-back player using a mere 70,000 neurons); it's simply not sufficiently well-organized.

If your answer to the above is "no way José", please say why. "It's not designed for that" is not sufficient; things do things they weren't designed to do all the time.

Comment author: Vladimir_Nesov 05 September 2009 01:52:07PM * 0 points [-]

You don't need to bypass the safeties to do better. What you need is not a bigger hammer with which to change the brain, but a better idea of what to change, and what to change it to.

But you do need a bigger hammer as well. And that bigger hammer is dangerous.

Comment author: pjeby 05 September 2009 05:25:04PM 1 point [-]

But you do need a bigger hammer as well.

For what, specifically?

continue this thread »

Comment author: rhollerith_dot_com comment score below threshold [+] (0 children)

Comment author: Psy-Kosh 04 September 2009 12:35:20AM * 0 points [-]

Ooooh, okay then. That makes sense.

Hrm... given though your suggested scenario, why the need to start with looking for other volunteers? ie, if the initial person is willing to be modified under the relevant constraints, why not just, well, spawn off another instance of themselves, one the modifier and one the modifiee?

Psy-Kosh comments on The Sword of Good - Less Wrong

Article Navigation

Comments (292)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (292)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login