= 27d567bc2d48d9f40e9ccc20ea370b15)
JamesAndrix comments on Dreams of AIXI - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (145)
What does physics say about a single entity doing an intelligence explosion?
In the event of alien competition, our AI should weigh our options according to our value system.
Under what conditions will a superintelligence alter it's value system except in accordance with it's value system? Where does that motivation come from? If a superintelligence prefers it's values to be something else, why would it not change it's preferences?
If it does, and the new preferences cause it to again want to modify its preferences, and so on again, will some sets of initial preferences yield stable preferences? or must all agents have preferences that would cause them to modify their preferences if possible?
Science lets us modify our beliefs in an organized and more reliable way. It could in principle be the case that a scientific investigation leads you to the conclusion that we should use other different rules, because they would be even better than what we now call science. But we would use science to get there, or whatever our CURRENT learning method is. Likewise we should change our values according to what we currently value and know.
We should design AI such that if it determines that we would consider 'personal uniqueness' extremely important if we were superintelligent, then it will strongly avoid any highly accurate simulations, even if that costs some accuracy. (Unless outweighed by the importance of the problem it's trying to solve.)
If we DON'T design AI this way, then it will do many things we wouldn't want, well beyond our current beliefs about simulations.
A great deal. I discussed this in another thread, but one of the constraints of physics tells us that the maximum computational efficiency of a system, and thus its intelligence, is inversely proportional to its size (radius/volume). So its extraordinarily unlikely, near zero probability i'd say, that you'll have some big global distributed brain with a single thread of consciousness - the speed of light just kills that. The 'entity' would need to be a community (which certainly still can be coordinated entities, but its fundamentally different than a single unified thread of thought).
Moreover, I believe the likely scenario is evolutionary:
The evolution of AGI's will follow a progression that goes from simple AGI minds (like those we have now in some robots) up to increasingly complex variants and finally up to human-equivalent and human-surpassing. But all throughout that time period there will be many individual AGI's, created by different teams, companies, and even nations, thinking in different languages, created for various purposes, and nothing like a single global AI mind. And these AGI's will be competing with both themselves and humans - economically.
I agree with most of the rest of your track of thought - we modify our beliefs and values according to our current beliefs and values. But as I said earlier, its not static. Its also not even predictable. Its not even possible, in principle, to fully predict your own future state. This to me, is perhaps the final nail in the coffin for any 'perfect' self-modifying FAI theory.
Moreover, I also find it highly unlikely that we will ever be able to create a human level AGI with any degree of pre-determined reliability about its goal system whatsoever.
I find it more likely that the AGI's we end up creating will have to learn ethics, morality, etc - their goal systems can not be hard coded, and whether they turn out friendly or not is entirely dependent on what they are taught and how they develop.
In other words, friendliness is not an inherent property of AGI designs - its not something you can design in to the algorithms itself. The algorithms for an AGI give you something like an infant brain - its just a canvas, its not even a mind yet.
On what basis will they learn? You're still starting out with an initial value system and process for changing the value system, even if the value system is empty. There is no reason to think that a given preference-modifier will match humanity's. Why will they find "Because that hurts me" to be a valid point? Why will they return kindness with kindness?
You say the goal systems can't be designed in, why not?
It may be the case that we will have a wide range of semifriendly subhuman or even near human AGI's. But when we get a superhuman AGI that is smart enough to program better AGI, why can it not do that on it's own?
I am positive that 'single entity' should not have mapped to 'big distributed global brain'.
But I also think an AIXI like algorithm would be easy to parallelize and make globally distributed, and it still maximizes a single reward function.
They will have to learn by amassing a huge amount of observations and interactions, just as human infants do, and just as general agents do in AI theory (such as AIXI).
Human brains are complex, but very little of that complexity is actually precoded in the DNA. For humans values, morals, and high level goals are all learned knowledge, and have varied tremendously over time and cultures.
Well, if you raised the AI as such, it would.
Consider that a necessary precursor of of following the strategy 'returning kindness with kindness' is understanding what kindness itself actually is. If you actually map out that word, you need a pretty large vocabulary to understand it, and eventually that vocabulary rests on grounded verbs and nouns. And to understand those, they must be grounded on a vast pyramid of statistical associations acquired from sensorimotor interaction (unsupervised learning .. aka experience). You can't program in this knowledge. There's just too much of it.
From my understanding of the brain, just about every concept has (or can potentially have) associated hidden emotional context: "rightness" and "wrongness", and those concepts: good, bad, yes, no, are some of the earliest grounded concepts, and the entire moral compass is not something you add later, but is concomitant with early development and language acquisition.
Will our AI's have to use such a system as well?
I'm not certain, but it may be such a nifty, powerful trick, that we end up using it anyway. And even if there is another way to do that is still efficient, it may be that you can't really understand human languages unless you also understand the complex web of value. If nothing else, this approach certainly gives you control over the developing AI's value system. It appears for human minds the value system is immensely complex - it is intertwined at a fundamental level with the entire knowledge base - and is inherently memetic in nature.
What is an AGI? It is a computer system (hardware), some algorithms/code (which actually is always eventually better to encode directly in hardware - 1000X performance increase), and data (learned knowledge). The mind part - all the qualities of importance, comes solely from the data.
So the 'programming' of the AI is not that distinguishable from the hardware design. I think AGI's will speed this up, but not nearly as dramatically as people here think. Remember humans don't design new computers anymore anyway. Specialized simulation software does the heavy lifting - and it is already the bottleneck. An AGI would not be better than this specialized software at its task (generalized vs specialized). It will be able to improve it some almost certainly, but only to the theoretical limits, and we are probably already close enough to them that this improvement will be minor.
AGI's will have a speedup effect on moore's law, but I wouldn't be surprised if this just ends up compensating for the increased difficulty going forward as we approach quantum limits and molecular computing.
In any case, we are simulation bound already and each new generation of processors designs (through simulation) the next. The 'FOOM' has already begun - it began decades ago.
Well I'm pretty certain that AIXI like algorithms aren't going to be directly useful - perhaps not ever, only more as a sort of endpoint on the map.
But that's beside the point.
If you actually use even a more practical form of that general model - a single distributed AI with a single reward function and decision system, I can show you how terribly that scales. Your distributed AI with a million PC's is likely to be less intelligent than a single AI running on tightly integrated workstation class machine with just say 100x the performance of one of your PC nodes. The bandwidth and the latency issues are just that extreme.
If concepts like kindness are learned with language and depend on a hidden emotional context, then where are the emotions learned?
What is the AI's motivation? This is related to the is-ought problem: no input will affect the AI's preferences unless there is something already in the AI that reacts to that input that way.
If software were doing the heavy lifting, then it would require no particular cleverness to be a microprocessor design engineer.
The algorithm plays a huge role in how powerful the intelligence will be, even if it is implemented in silicon.
People might not make most of the choices in laying out chips, but we do almost all of the algorithm creation, and that is where you get really big gains. see Deep Fritz vs. Deep Blue. Better algorithms can let you cut out a billion tests and output the right answer on the first try, or find a solution you just would not have found with your old algorithm.
Software didn't invent out of order execution. It just made sure that the design actually worked.
As for the distributed AI: I was thinking of nodes that were capable of running and evaluating whole simulations, or other large chunks of work. (Though I think superintelligence itself doesn't require more than a single PC.)
In any case, why couldn't your supercomputer foom?
I think this is an open question, but certainly one approach is to follow the brain's lead and make a system that learns its ethics and high level goals dynamically, through learning.
In that type of design, the initial motivation gets imprinting queues from the parents.
Oh of course, but I was just pointing out that after a certain amount of research work in a domain, your algorithms converge on some asymptotic limit for the hardware. There is nothing even close to unlimited gains purely in software.
And the rate of hardware improvement is limited now by speed of simulation on current hardware, and AGI can't dramatically improve that.
Yes, of course. Although as a side note we are moving away from out of order execution at this point.
Because FOOM is just exponential growth, and in that case FOOM is already under way. It could 'hyper-FOOM', but the best an AGI can do is to optimize its brain algorithms down to the asymptotic limits of its hardware, and then it has to wait with everyone else until all the complex simulations complete and the next generation of chips come out.
Now, all that being said, I do believe we will see a huge burst of rapid progress after the first human AGI is built, but not because that one AGI is going to foom by itself.
The first human-level AGI's will probably be running on GPUs or something similar, and once they are proven and have economic value, there will be this huge rush to encode those algorithms directly in to hardware and thus make them hundreds of times faster.
So I think from the first real-time human-level AGI it could go quickly to 10 to 100X AGI (in speed) in just a few years, along with lesser gains in memory and other IQ measures.
This seems like a non-answer to me.
You can't just say 'learning' as if all possible minds will learn the same things from the same input, and internalize the same values from it.
There is something you have to hardcode to get it to adopt any values at all.
Well, what is that limit?
It seems to me that an imaginary perfectly efficient algorithm would read process and output data as fast as the processor could shuffle the bits around, which is probably far faster than it could exchange data with the outside world.
Even if we take that down 1000x becsaue this is an algorithm that's doing actual thinking, you're looking at an easy couple of million bytes per second. And that's superintelligently optimized structured output based on preprocessed efficient input. Because this is AGI, we don't need to count in say, raw video bandwidth, because that can be preprocessed by a system that is not generally intelligent.
So a conservatively low upper limit for my PC's intelligence is outputting a million bytes per second of compressed poetry, or viral genomes, or viral genomes that write poetry.
If the first Superhuman AGI is only superhuman by an order of magnitude or so, or must run on a vastly more powerful system, then you can bet that it's algorithms are many orders of magnitude less efficient than they could be.
No.
Why couldn't your supercomputer AGI enter into a growth phase higher than exponential?
Example: If not-too-bright but technological aliens saw us take a slow general purpose computer, and then make a chip that worked 100 times faster, but they didn't know how to put algorithms on a chip, then it would look like our technology got 1000 times better really quickly. But that's just because they didn't already know the trick. If they learned the trick, they could make some of their dedicated software systems work 1000 times faster.
"Convert algorithm to silicon." is just one procedure for speeding things up that an agent can do, or not yet know how to do. You know it's possible, and a superintelligence would figure it out, but how do you rule out a superintelligence figureing out twelve trick like that, which each provide a 1000x speedup. In it's first calendar month?
Yes, you have to hardcode 'something', but that doesn't exactly narrow down the field much. Brains have some emotional context circuitry for reinforcing some simple behaviors (primary drives, pain avoidance, etc), but in humans these are increasingly supplanted and to some extent overridden by learned beliefs in the cortex. Human values are thus highly malleable - socially programmable. So my comment was "this is one approach - hardcode very little, and have all the values acquired later during development".
Unfortunately, we need to be a little more specific than imaginary algorithms.
Computational complexity theory is the branch of computer science that deals with the computational costs of different algorithms, and specifically the most optimal possible solutions.
Universal intelligence is such a problem. AIXI is an investigation into optimal universal intelligence in terms of the upper limits of intelligence (the most intelligent possible agent), but while interesting, it shows that the most intelligent agent is unusably slow.
Taking a different route, we know that a universal intelligence can never do better in any specific domain than the best known algorithm for that domain. For example, an AGI playing chess could do no better than just pausing its AGI algorithm (pausing its mind completely) and instead running the optimal chess algorithm (assuming that the AGI is running as a simulation on general hardware instead of faster special-purpose AGI hardware).
So there is probably an optimal unbiased learning algorithm, which is the core building block of a practical AGI. We don't know for sure what that algorithm is yet, but if you survey the field, there are several interesting results. The first thing you'll see is that we have a variety of hierarchical deep learning algorithms now that are all pretty good, some appear to be slightly better for certain domains, but there is not atm a clear universal winner. Also, the mammalian cortex uses something like this. More importantly, there is alot of recent research, but no massive breakthroughs - the big improvements are coming from simple optimization and massive datasets, not fancier algorithms. This is not definite proof, but it looks like we are approaching some sort of bound for learning algorithms - at least at the lower levels.
There is not some huge space of possible improvements, thats just not how computer science works. When you discover quicksort and radix sort, you are done with serial sorting algorithms. And then you find the optimal parallel variants, and sorting is solved. There are no possible improvements past that point.
Computer science is not like moore's law at all. Its more like physics. There's only so much knowledge, and so many breakthroughs, and at this point alot of it honestly is already solved.
So its just pure naivety to think that AGI will lead to some radical recursive breakthrough in software. poppycock. Its reasonably likely humans will have narrowed in on the optimal learning algorithms by the time AGI comes around. Further improvements will be small optimizations for particular hardware architectures - but thats really not much different at all then hardware design itself, and eventually you want to just burn the universal learning algorithms into the hardware (as the brain does).
Hardware is quite different, and there is a huge train of future improvements there. But AGI's impact there will be limited by computer speeds! Because you need regular computers running compilers and simulators to build new programs and new hardware. So AGI can speed Moore's Law up some, but not dramatically - an AGI that thought 1000x faster than a human would just spend 1000x longer waiting for its code to compile.
I am a software engineer, and I spend probably about 30-50% of my day waiting on computers (compiling, transferring, etc). And I only think at human speeds.
AGI's will soon have a massive speed advantage, but ironically they will probably leverage that to become best selling authors, do theoretical physics and math, and non-engineering work in general where you don't need alot of computation.
Say you had an AGI that thought 10x faster. It would read and quickly learn everything about its own AGI design, software, etc etc. It would get a good idea of how much optimization slack there was in its design and come up with a bunch of ideas. It could even write the code really fast. But unfortunately it would still have to compile it and test it (adding extra complexity in that this is its brain we are talking about).
Anyway, it would only be able to get small gains from optimizing its software - unless you assume the human programmers were idiots. Maybe a 2x speed gain or something - we are just throwing numbers out, but we have a huge experience with real-time software on fixed hardware in say the video game industry (and other industries) and this asymptotic wall is real, and complexity theory is solid.
Big gains necessarily must come from hardware improvements. This is just how software works - we find optimal algorithms and use them, and further improvement without increasing the hardware hits an asymptotic wall. You spend a few years and you get something 3x better, spend 100 more and you get another 50%, and spend 1000 more and get another 30% and so on.
EDIT: After saying all this, I do want to reiterate that I think there could be a quick (even FOOMish) transition from the first AGIs to AGI's that are 100-1000x or so faster thinking, but the constraint on progress will quickly be the speed of regular computers running all the software you need to do anything in the modern era. Specialized software already does much of the heavy lifting in engineering, and will do even more of it by the time AGI arrives.
Hardcode very little?
What is the information content of what an infant feels when it is fed after being hungry?
I'm not trying to narrow the feild, the feild is always narrowed to whatever learning system an agent actually uses. In humans, the system that learns new values is not generic
Using a 'generic' value learning system will give you an entity that learns morality in an alien way. I cannot begin to guess what it would learn to want.
I'd like to table the intelligence explosion portion of this discussion, I think we agree that an AI or group of AI's could quickly grow powerful enough that they could take over, if that's what they decided to do. So establishing their values is important regardless of precisely how powerful they are.
Yes. The information in the genome, and the brain structure coding subset in particular, is a tiny tiny portion of the information in an adult brain.
An infant brain is mainly an empty canvas (randomized synaptic connections from which learning will later literally carve out a mind) combined with some much simpler, much older basic drives and a simpler control system - the old brain - that descends back to the era of reptiles or earlier.
That depends on what you mean by 'values'. If you mean linguistic concepts such as values, morality, kindness, non-cannibalism, etc etc, then yes, these are learned by the cortex, and the cortex is generic. There is a vast weight of evidence for almost overly generic learning in the cortex.
Not at all. To learn alien morality, it would have to either invent alien morality from scratch, or be taught alien morality from aliens. Morality is a set of complex memetic linguistic patterns that have evolved over long periods of time. Morality is not coded in the genome and it does not spontaneously generate.
Thats not to say that there are no genetic tweaks to the space of human morality - but any such understanding based on genetic factors must also factor in complex cultural adaptations.
For example, the Aztecs believed human sacrifice was noble and good. Many Spaniards truly believed that the Aztecs were not only inhuman, but actually worse than human - actively evil, and truly believed that they were righteous in converting, conquering, or eliminating them.
This mindspace is not coded in the genome.
Agreed.