Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

spindizzy2 comments on That Alien Message - Less Wrong

111 Post author: Eliezer_Yudkowsky 22 May 2008 05:55AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (164)

Sort By: Old

You are viewing a single comment's thread.

Comment author: spindizzy2 22 May 2008 06:23:59PM 1 point [-]

Eliezer, you must have lowered your intellectual level because these days I can understand your posts again.

You talk about the friendliness problem as if it can be solved separately from the problem of building an AGI, and in anticipation of that event. I mean that you want to delay the creation of an AGI until friendliness is fully understood. Is that right?

Suppose that we had needed to build jet-planes without ever passing through the stage of propeller-based planes, or if we had needed to build modern computers without first building calculators, 8-bit machines etc. Do you think that would be possible? (It's not a rhetorical question... I really don't know the answer).

It seems to me that if we ever build an AGI, there will be many mistakes made along the way. Any traps waiting for us will certainly be triggered.

Perhaps this will all seem clearer when we all have 140 IQ's. Get to work, Razib! :)

Comment author: pnrjulius 09 April 2012 04:49:11AM 0 points [-]

Interesting. And to use a more poignant example: Could we have invented fusion power if we had never invented the atomic bomb?

It does seem plausible to say that technological growth is necessarily (or if not necessarily, at least typically) incremental; it doesn't proceed in huge leaps and breakthroughs, but in slow accumulations of tinkering.

Of more concern is the equally plausible inference that tinkering requires making mistakes, and as our technology improves, the mistakes will have larger and larger stakes on which to miss.