bentarm comments on Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It) - Less Wrong

32 Post author: ciphergoth 30 October 2010 09:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (432)

You are viewing a single comment's thread. Show more comments above.

Comment author: bentarm 02 November 2010 12:17:22PM 0 points [-]

Please give an example of why the AGI should co-operate with something that cannot do anything the AGI itself cannot.

A sufficiently clever AI should understand Comparative Advantage

Comment author: Vladimir_Nesov 02 November 2010 12:56:36PM *  13 points [-]

Comparative advantage explains how to make use of inefficient agents, so that ignoring them is a worse option. But if you can convert them into something else, you are no longer comparing the gain from trading with them to indifference of ignoring them, you are comparing the gain from trading with them to the gain from converting them. And if they can be cheaply converted into something much more efficient than they are, converting them is the winning move. This is a move largely not available to the present society, hence its absence is a reasonable assumption for now but one that breaks when you consider indifferent smart AGI.

Comment author: JGWeissman 02 November 2010 04:59:59PM *  1 point [-]

The law of comparative advantage relies on some implicit assumptions that are not likely to hold between a superintelligence and humans:

The transactions costs must be small enough not to negate the gains from trade. A superintelligence may require more resources to issue a trade request to slow thinking humans and to receive the result, while possibly letting processes idle while waiting for the result, than to just do it itself.

Your trading partner must not have the option of building a more desirable trading partner out of your component parts. A superintelligence could get more productivity of atoms arranged as an extension of itself than atoms arranged as humans. (ETA: See Nesov's comment.)

Comment author: pjeby 02 November 2010 04:08:16PM 1 point [-]

A sufficiently clever AI should understand Comparative Advantage

And a sufficiently clever human should realize that clever humans can and do routinely increase the efficiencies of their industry enough to shift the comparative advantage.

It really doesn't take that much human-level intelligence to change how things are done -- all it takes is a lack of attachment to the current ways.

And that's perhaps the biggest "natural resource" an AI has: the lack of status quo bias.

Comment author: bentarm 02 November 2010 04:51:31PM *  0 points [-]

And a sufficiently clever human should realize that clever humans can and do routinely increase the efficiencies of their industry enough to shift the comparative advantage.

I'm not sure I understand what "shift the comparative advantage" could mean, and I have no idea why this is supposed to be a response to my point.

Maybe I didn't make my point clearly enough. My contention is that even if an AI is better at absolutely everything than a human being, it could still be better off trading with human beings for certain goods, for the simple reason that it can't do everything, and in such a scenario both human beings and the AI would get gains from trade.

As Nesov points out, if the AI has the option of, say, converting human beings into computational substrate and using them to simulate new versions of itself, then this ceases to be relevant.

Comment author: Vladimir_Nesov 02 November 2010 04:26:10PM *  0 points [-]

And a sufficiently clever human should realize that clever humans can and do routinely increase the efficiencies of their industry enough to shift the comparative advantage.

I don't understand what are you arguing for. That people become better off doing something different, doesn't necessarily imply that they become obsolete, or even that they can't continue doing the less-efficient thing.