= 27d567bc2d48d9f40e9ccc20ea370b15)
bentarm comments on Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It) - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (432)
Please give an example of why the AGI should co-operate with something that cannot do anything the AGI itself cannot.
Right. E. coli don't offer us anything we can't do for ourselves, that we can't just whip up a batch of E. coli for on demand.
If I'm a god, what would I need a human for? If I need humans, I can just make some. Better still, I could replace them with something more efficient that doesn't complain or rebel.
The fundamental flaw in your reasoning here is that you keep trying to construct paths through probability space that could support your hypothesis, but ONLY if you had presented some evidence for singling out that hypothesis in the first place!
It's like you're a murder investigator opening up the phonebook to a random place and saying, "well, we haven't ruled out the possibility that this guy did it", and when people quite reasonably point out that there is no connection between that random guy and the murder, you reply, "yeah, but I just called this guy, and he has no alibi." (That is, you're ignoring the fact that a huge number of people in that phonebook will also have no alibi, so your "evidence" isn't actually increasing the expected probability that that guy did it.)
And that's why you're getting so many downvotes: in LW terms, you are failing basic reasoning.
But that is not a shameful thing: any normal human being fails basic reasoning, by default, in exactly the same way. Our brains simply aren't built to do reasoning: they're built to argue, by finding the most persuasive evidence that supports our pre-existing beliefs and hypotheses, rather than trying to find out what is true.
When I first got here, I argued for some of my pet hypotheses in the exact same way, although I was righteously certain that I was not doing such a thing. It took a long time before I really "got" Bayesian reasoning sufficiently to understand what I was doing wrong, and before that, I couldn't have said here what you were doing wrong either.
A sufficiently clever AI should understand Comparative Advantage
Comparative advantage explains how to make use of inefficient agents, so that ignoring them is a worse option. But if you can convert them into something else, you are no longer comparing the gain from trading with them to indifference of ignoring them, you are comparing the gain from trading with them to the gain from converting them. And if they can be cheaply converted into something much more efficient than they are, converting them is the winning move. This is a move largely not available to the present society, hence its absence is a reasonable assumption for now but one that breaks when you consider indifferent smart AGI.
The law of comparative advantage relies on some implicit assumptions that are not likely to hold between a superintelligence and humans:
The transactions costs must be small enough not to negate the gains from trade. A superintelligence may require more resources to issue a trade request to slow thinking humans and to receive the result, while possibly letting processes idle while waiting for the result, than to just do it itself.
Your trading partner must not have the option of building a more desirable trading partner out of your component parts. A superintelligence could get more productivity of atoms arranged as an extension of itself than atoms arranged as humans. (ETA: See Nesov's comment.)
And a sufficiently clever human should realize that clever humans can and do routinely increase the efficiencies of their industry enough to shift the comparative advantage.
It really doesn't take that much human-level intelligence to change how things are done -- all it takes is a lack of attachment to the current ways.
And that's perhaps the biggest "natural resource" an AI has: the lack of status quo bias.
I'm not sure I understand what "shift the comparative advantage" could mean, and I have no idea why this is supposed to be a response to my point.
Maybe I didn't make my point clearly enough. My contention is that even if an AI is better at absolutely everything than a human being, it could still be better off trading with human beings for certain goods, for the simple reason that it can't do everything, and in such a scenario both human beings and the AI would get gains from trade.
As Nesov points out, if the AI has the option of, say, converting human beings into computational substrate and using them to simulate new versions of itself, then this ceases to be relevant.
I don't understand what are you arguing for. That people become better off doing something different, doesn't necessarily imply that they become obsolete, or even that they can't continue doing the less-efficient thing.