Students asked to defend AGI danger update in favor of AGI riskiness

lukeprog

Students asked to defend AGI danger update in favor of AGI riskiness — LessWrong

Comment Permalink

Some computer programs crash - just as some possible superintelligences would kill alll humans.

However, the behavior of a computer program chosen at random tells you very little about how an actual real-world computer program will behave - since computer programs are typically produced by selection processes performed by intelligent agents.

The "for almost any goals" argument is bunk.

Eugine_Nier15y20

Some computer programs crash - just as some possible superintelligences would kill all humans.

No most computer programs crash, it's just that most people never see them because said programs are repeatedly tested and modified until they no longer crash before being shown to people (this process is called "debugging"). With a self-modifying AI this is a lot harder to do.

1Logos0115y

A statement which we can derive from the simple fact that the mere existence of general intelligence (apes) does not result automatically in catastrophe. I wonder how long it'll take before people catch onto the notion that artificial "dumbness" is in many ways a more interesting field than artificial "intelligence"? (As in, how much could an AGI no smarter than a dog, but hooked into expert systems similar to Watson, do?)

2J_Taylor15y

I am not entirely sure I disagree with you. However, I am having difficulty modeling you. "Achieving a goal" seems to mean, for our purposes, something along the lines of "Bringing about a world-state." Most possible world-states do not involve human existence. Thus, it seems that for most possible goals, achieving a goal entails human extinction. However, your mention of computer programs being produced by intelligent agents is interesting. Are you implying that most AGI's (assume these intelligences can go FOOM) would not result in human extinction? If this is not what you were implying, I apologize for modeling you poorly. If this is what you were implying, I would like to indicate that this post was non-hostile.

See in context

7

Students asked to defend AGI danger update in favor of AGI riskiness

7

7

7

Students asked to defend AGI danger update in favor of AGI riskiness

7

7