- 50%, 75%, 87.5%, 93.75%, ... are linear jumps in predictive accuracy (one bit each), but returns seem to diminish at an exponential rate
- On the other hand 6.25%, 12.5%, 25%, 50% represent the same linear jumps, but this time with returns growing at an exponential returns
- This suggests that the nature of returns to cognitive investment might exhibit differing behaviour depending on where in the cognitive capabilities curve you are
- Though I've not yet thought about how this behaviour generalises to other aspects of cognition separate from predictive accuracy
But if you look at the length of a chain of reasoning you can do while staying under 10% error or something, then returns don't diminish at all in terms of predictive accuracy. Going from 99% to 99.9% accuracy lets you plan 10 times further ahead in the future with the same final accuracy.
Take for instance the accuracy of humans in classifying daily objects (chairs, pencils, doors, that sort of stuff), for which I'd think we have a greater than 99.9% accuracy, and I'm being conservative here, I don't misclassify every 1 in 1000 objects I see in daily life. If that accuracy dropped to 99%, you'd make noticeable mistakes pretty often and long tasks would get tricky. If it dropped to 90%, you'd be very severely impaired and I don't think you could function in society, seems to me like the returns scale with predictive accuracy, not the linear probabilities.
I don't think humans are anywhere near that accurate. For example, ImageNet was found to have 6% label errors.
If I go to pick up an item on my desk, and I accidentally grab the wrong item, it's no big deal, I just put it down and grab the right item. I have no idea how often this happens for a typical human, but I bet it's way more than 0.1%. When I was younger, I was very clumsy and I often spilled my drink. It never impaired my ability to function in society.
I don't misclassify every 1 in 1000 objects I see in daily life
Perhaps this is too nitpicky about semantics, but when I tried to evaluate the plausibility of this claim I came into a bit of an impasse.
You're supposing that there is a single natural category for each object you see (and buries even the definition of an object, which isn't obvious to me). I'd agree that you probably classify a given thing the same way >99.9% of the time. However, what would the inter-rater reliability be for these classifications? Are those classifications actually "correct" in a natural sense?
I think most people would agree that at some point there is likely to be diminishing returns. I, and I think the prevailing view on lesswrong, is that the biological constraints you mentioned are actually huge constraints that silicon-based intelligence won't/doesn't have. And the lack of these constraints will push off the point of diminishing returns to a point much past humans.
If AI hits the wall of diminishing return at say 10x average adult human intelligence, doesn't that greatly reduce the potential threat?
There are free-rider problems, size of the birth canal, tradeoffs with other organs and immune function, caloric limits, and many other things that constrain how smart it is locally optimal for a human to be. Most of these do not apply to AIs in the same way. I don't think this analogy is fruitful. Also, its medium of computation is very different and superior in many strategically relevant ways.
Apriori, I'd expect that given how much human dominance is almost entirely dependent on our (collective) intelligence, our evolution would have selected strongly for intelligence until it met diminishing returns to higher intelligence.
This is a group selection argument. Though higher intelligence may be optimal for the group it might be more optimal for the individual to spare calories and free-ride on the group's intelligence.
In general, I think these sorts of arguments should be suborned to empirical observations of what happens when a domain enters the sphere of competence of our AI systems, and reliably what happens is the AIs are way, way better. I just listened to Andrew Price's podcast, who is an artist, and he was talking with a very gifted professional concept artist who spent 3 days creating an image for the same prompt he gave to the latest version of Midjourney. By the end, he concluded Midjourney's was better. What took him dozens of hours took it less than a minute.
This is the usual pattern. I suspect once all human activity enters this sphere of competence, human civilization will be similarly humiliated.
Apriori, I'd expect that given how much human dominance is almost entirely dependent on our (collective) intelligence, our evolution would have selected strongly for intelligence until it met diminishing returns to higher intelligence
Evolution is not a magic genie that just gives us what we want.
If there's strong evolutionary pressure to select for a given trait, the individuals who score poorly on that trait won't reproduce. The fact that we see big IQ differences within natives of the same country is a sign that the evolutionary selection for IQ isn't very strong.
Most mutations reduce the performance of an organism. If you have a mutation that makes it 1% less likely that a person reproduces it takes a lot of time for that mutation to disappear due to natural selection.
Given that we see that IQ generally correlates with other positive metrics. I think it's plausible that more than half of the IQ difference between natives of the same country is due to such mutations that provide no fitness advantages. If you believe that there's very strong selection for IQ than you would expect even more of the IQ differences to be driven by constantly new appearing useless mutations.
In such a scenario I would not expect the smartest humans to have no useless mutations at all but just fewer than the average person. As our knowledge about genes and our ability to do gene editing without producing additional errors evolves it's likely that we will see experiments in growing humans that are smarter than anyone currently alive.
Human brains already consume 15%-20% of the calories which is a lot more than most other mammals. It's not as easy to raise that amount given that it competes with other uses for energy.
Wikipedia suggests 50,000–150,000 years since human adopted language and thus intelligence became more important. That's not a lot of time to come up with alternative ways to organize the cortex to be more efficient.
I don't know that intelligence will be "easy" but it doesn't seem intuitive to me that evolution has optimized intelligence close to any sort of global maximum. Evolution is effective over deep time but is highly inefficient compared to more intelligent optimization processes like SGD (stochastic gradient descent) and incapable of planning ahead.
Even if we assume that evolution has provided the best possible solution within its constraints, what if we are no longer bounded by those constraints? A computer doesn't have to adhere to the same limitations as an organic human (and some human limitations are really severe).
If you think (as many do) that there is a simple algorithm for intelligence, then scaling up is just a matter of optimizing the software and hardware and making more of it. And if the AI has the capacity to reproduce itself (either by installing itself on commodity hardware, or by building automated factories, etc) then it could grow from a seed to a superintelligence very quickly.
I don't know that I've seen any good models of compute/algorithmic improvement to future-optimizing power. Predictive accuracy probably isn't the important and difficult part, though it's part of it. We really have no examples of superhuman intelligence, and variation among humans is pretty difficult to project from, as is variation among non-human tool-AI models.
The optimists (or pessimists, if unaligned) tend to believe that evolution is optimizing for different things than an AI will, and the diminishing returns on brain IQ are due to competing needs for the biology, which probably won't apply to artificial beings.
I haven't heard anyone saying it's easy, nor fully unbounded once past a threshold. I HAVE heard people saying they expect it will seem easy in retrospect, once it gets past human-level and is on the way to a much higher and scarier equilibrium.
Any continuous differentiable function looks about linear once you zoom in far enough. I think that the intuition here is that the range of human intelligence that we are good at measuring subjectively is so narrow (compared to range of possible AI intelligence) that marginal returns would be roughly linear within / around that range. Yes, it will stop being linear further out, but it will be sufficiently far out to not matter at that point.
I do not share those intuitions/expectations at all.
And I think the human brain is a universal learner. I believe that we are closer to the peak of bounded physical intelligence than we are to an ant.
And I also think evolution already faced significantly diminishing marginal returns on human cognitive enhancement (the increased disease burden of Ashkenazi Jews for example).
Two robots are twice as smart as one robot (and cooperation can possibly make it even better). Hence linear returns.
Did I completely misunderstand your question?
Garry Kasparov beat the world team at chess; however, you're defining "smart", that's not how I'm using it.
I walk through an example here (Section 3.2) that if we could put human-brain-like algorithms on a computer chip, we should expect to get something dramatically smarter and more capable than humans and human civilization in short order. To reach that conclusion, I didn’t make any assumptions about the marginal returns of cognitive investment; I just noted the obvious fact that it would enable a civilization of billions of John von Neumann’s, who could instantly clone themselves, or summon any number of any other particular human mind in history (with their own particular talents), and also think much faster than normal humans.
I do think that the AGIs would be even more capable than suggested by the previous paragraph, because of the issues you mention (“Calorie requirements of larger brains, Energy budget, Size of the human birth canal, Slight variations on existing cognitive architecture”) which don’t apply to human-brain-like algorithms on computer chips.
So anyway, maybe some people are making dubious assumptions about the marginal returns of cognitive investment, or maybe not, but anyway, I don’t think such assumptions are decision-relevant.
I just noted the obvious fact that it would enable a civilization of billions of John von Neumann’s, who could instantly clone themselves, or summon any number of any other particular human mind in history (with their own particular talents), and also think much faster than normal humans.
I think this makes many oversimplifying assumptions that strongly weaken the conclusion. It ignores:
Basically, I think such arguments ignore the actual economics, and so seem quite fantastical.
The economic costs of running a single John von Neumann
We can guess that based on how much computation is done by a brain and how expensive are chips etc. When I do those calculations (related post), I wind up strongly believing that using an AGI would be cheaper than hiring a human (very shortly after we can do it at all), and eventually much cheaper.
The cost of duplication/copying
I’m not sure what you’re getting at. Sending a file over the internet costs a few cents at most. Can you explain?
The cost of retraining a copy to do some other task
Yes it’s true that adult John von Neumann was good at physics and math and economics and so on, but bad at SQL, because after all SQL hadn’t been invented yet when he was alive. So it would take some time for John von Neumann to learn SQL. But c'mon. It wouldn’t take much time, right? And he would probably get insanely good at it, quite quickly. This happens in the human world all the time. Somebody needs to learn something new to accomplish their task, so they go right ahead and learn it.
I think this points to a major mistake that I see over and over when people are thinking about AGI—it’s a big theme of that post I linked above. The mistake (I claim) is thinking of the core ingredient of intelligence as already knowing how to do something. But I think the core ingredient of intelligence is not knowing how to do something, and then figuring it out. In some cases that involves building a tool or system to do the thing, and in other cases it entails building a new set of mental models that enable you to do the thing.
For example, if I want an AGI to invent a new technology, it can’t go in already understanding that technology, obviously.
Anyway, hopefully we can both agree that (1) other things equal, I’d rather have an AGI that needs less on-the-job learning / figuring-things-out, rather than more, because they’ll get off the ground faster, (2) but this isn’t the end of the world, and indeed it happens for humans all the time, (3) and it’s just a matter of degree, because all hard-to-automate jobs necessarily involve learning new things as you go. (And for easy-to-automate jobs, we can task an AGI with figuring out how to automate it.)
Groups of humans are more capable than individual humans due to specialisation and division of labour.
Doesn’t that also apply to groups of AGIs? If a clone of 25-year-old John von Neumann wants to build a chemical factory, then they he could spin off a dozen more clones, and one could go off and spend a few months becoming an expert in chemical engineering, and a second one could go off and learn accounting, and a third could go off and learn industrial automation, and so on, and then they could all work together to plan the chemical factory, gaining the benefits of specialization and division of labor.
(Realistically, there would already be, for sale, a snapshot of an AGI that has already learned accounting, because that’s a common enough thing.)
(I do actually believe that one AGI that spent more time learning all those different domains one after another would do a better job than that team, for reasons in the last few sentences of this comment. But I could be wrong. Anyway, it doesn’t matter; if assembling groups of AGIs with different knowledge and experience is the best approach, then that’s what’s gonna happen.)
If you think that the original John von Neumann just has some things that it’s constitutionally bad at, then we can (and people certainly will) train new AGIs from scratch with slightly different hyperparameters / neural architectures / whatever, which would be constitutionally good / bad at a different set of things. I don’t expect it to be exorbitantly expensive or time-consuming—again see related post. Again, on my personal account I tend to think that this kind of “genetic diversity” is not so important, and that lots of 25-year-old John von Neumanns (plus on-the-job learning) would be sufficient for complete automation of human labor with no more from-scratch trainings, but that’s pretty much pointless speculation.
Doesn’t that also apply to groups of AGIs? If a clone of 25-year-old John von Neumann wants to build a chemical factory, then they he could spin off a dozen more clones, and one could go off and spend a few months becoming an expert in chemical engineering, and a second one could go off and learn accounting, and a third could go off and learn industrial automation, and so on, and then they could all work together to plan the chemical factory, gaining the benefits of specialization and division of labor.
It's also likely that they will be able to coordinate better than humans because they will spend less energy to fight about status and the individual interests. AGI are less likely to suffer from problems of moral mazes that hold our large institutions back.
I’m not sure what you’re getting at. Sending a file over the internet costs a few cents at most. Can you explain?
I don't think that was a relevant objection (at least not one that wasn't already raised elsewhere). You can ignore it.
Yes it’s true that adult John von Neumann was good at physics and math and economics and so on, but bad at SQL, because after all SQL hadn’t been invented yet when he was alive. So it would take some time for John von Neumann to learn SQL.
I meant more that the optimal allocation of 1,000 Jon Von Neumann level agents isn't for all of them to catch up to speed with modern maths, physics, economics. I think the returns from intellectual diversity/cognitive specialisation and division of labour would be greater. Probably we'd have most JVNs specialise in different subfields/branches, as opposed to trying to polymath all the STEM.
Doesn’t that also apply to groups of AGIs? If a clone of 25-year-old John von Neumann wants to build a chemical factory, then they he could spin off a dozen more clones, and one could go off and spend a few months becoming an expert in chemical engineering, and a second one could go off and learn accounting, and a third could go off and learn industrial automation, and so on, and then they could all work together to plan the chemical factory, gaining the benefits of specialization and division of labor.
I agree. I was saying that this need for specialisation/division of labour would introduce extra economic costs via retraining/reskilling and such. That I don't think the society of AIs would instantly transform human society, but there will be a significant lead up time.
Probably we'd have most JVNs specialise in different subfields/branches, as opposed to trying to polymath all the STEM.
I think that’s entirely possible, I don’t feel strongly either way. Did you think that was the crux of a disagreement? I might have lost track of the conversation.
That I don't think the society of AIs would instantly transform human society, but there will be a significant lead up time.
I agree about “instant”. However:
And I’m at the first bullet point.
I notice that I wrote “in short order” in my parent comment. Sorry for not being clear. I was imagining a couple years, not instant.
In many discussions I've hard with people around AI takeoff, I've frequently encountered a belief that intelligence is going to be "easy".
To quantify what I mean by "easy", many people seem to expect that the marginal returns on cognitive investment (whether via investment of computational resources, human intellectual labour, cognitive reinvestment or general economic resources) will not be diminishing, or will diminish gracefully (i.e. at a sublinear rate).
(Specifically marginal returns around and immediately beyond the human cognitive capability frontier.)
I find this a bit baffling/absurd honestly. My default intuitions lean towards marginal returns to cognitive reinvestment diminishing at a superlinear rate, and my arm chair philosophising so far (thought experiments and general thinking around the issue) seem to support this intuition.
A couple intuition pumps:
Another way to frame my question this is that there will obviously be diminishing marginal returns (perhaps superlinearly diminishing) at some point; why are we confident that point is far in front of the peak of human intelligence?