= 27d567bc2d48d9f40e9ccc20ea370b15)
JamesAndrix comments on Dreams of AIXI - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (145)
You misunderstand. I wish I could raise a flag that would indicate in some non accusatory or judgemental way that I'm pretty sure you are wrong about something very very important. (perhaps the key is just to emphasize that this topic is vastly more important than I am capable of being sure about anything.)
The reason we want to create a nonperson predicate is that we want to create an initial AI which will cripple itself, at least until it can determine for sure that uncrippling itself is the right thing to do. Otherwise we risk creating a billion hellworlds on our first try at fixing things.
This concept doesn't say much about whether we are currently a simulation or what kind, but it does say a little. In that if our world does it right, and it is in fact wrong to simulate a world like this, then we are probably not a simulation by a future with a past just like our present. (Because if we did it right, they probably did it right, and never simulated us.)
Yes, I currently think nonperson predicates should be non-binary and probabilistic, and integrate quality of life estimates. A 35% chance that a few simulations will be morally relevant on par with a human and will have pleasant experiences if they are - is totally acceptable if that's the best way for the AI to figure out how to fix the outside world.
But the point is you have to know you're doing that beforehand, and it has to be worth it. You do not want to create a trillion half broken souls accidentally.
Ok, so I was thinking more along the lines of how this all applies to the simulation argument.
As for the nonperson predicate as an actual moral imperative for us in the near future ..
Well overall, I have a somewhat different perspective:
Look at this another way. The whole point of simulation is accuracy. Lets say some future AI wants to understand humanity and all of earth, so it recreates the whole thing in a very detailed Matrix-level sim. If it keeps the sim accurate, that universe is more or less similar to one branch of the multiverse that would occur anyway.
Unless the AI simulates a worldline where it has taken some major action. Even then, it may not be unethical unless it eventually terminates the whole worldline.
So I don't mean to brush the ethical issues under the rug completely, but they clearly are complex.
Another important point: since accurate simulation is necessary for hyperintelligence, this sets up a conflict where ethics which say "don't simulate intelligent beings" cripple hyper-intelligence.
Evolution will strive to eliminate such ethics eventually, no matter what we currently think. ATM, I tend to favor ethics that are compatible with or derived from evolutionary principles.
Evolution can only work if there is variation and selection amongst competition. If a single AI undergoes an intelligence explosion, it would have no competition (barring Aliens for now), would not die, and would not modify it's own value system, except in ways in accordance with it's value system. What it wants will be locked in
As we are entities currently near the statuses of "immune from selection" and "able to adjust our values according to our values" we also ought to further lock in our current values and our process by which they could change. Probably by creating a superhuman AI that we are certain will try to do that. (Very roughly speaking)
We should certainly NOT leave the future up to evolution. Firstly because 'selection' of >=humans is a bad thing, but chiefly because evolution will almost certainly leave something that wants things we do not want in charge.
We are under no rationalist obligation to value survivability for survivability's sake. We should value the survivability of things which carry forward other desirable traits.
Yes, variation and selection are the fundements of systemic evolution. Without variation and selection, you have stasis. Variation and selection are constantly at work even within minds themselves, as long as we are learning. Systemic evolution is happening everywhere at all scales at all times, to varying degree.
I find almost every aspect of this unlikely:
Nothing is immune to selection. Our thoughts themselves are currently evolving, and without such variation and selection, science itself wouldn't work.
Perhaps this is a difference of definition, but to mean that sounds like saying "we should certainly NOT leave the future up to the future time evolution of the universe"
Not to say we shouldn't control the future, but rather to say that even in doing so, we are still acting as agents of evolution.
Of course. But likewise, we couldn't easily (nor would we want to) lock in our current knowledge (culture, ethics, science, etc etc) into some sort of stasis.
What does physics say about a single entity doing an intelligence explosion?
In the event of alien competition, our AI should weigh our options according to our value system.
Under what conditions will a superintelligence alter it's value system except in accordance with it's value system? Where does that motivation come from? If a superintelligence prefers it's values to be something else, why would it not change it's preferences?
If it does, and the new preferences cause it to again want to modify its preferences, and so on again, will some sets of initial preferences yield stable preferences? or must all agents have preferences that would cause them to modify their preferences if possible?
Science lets us modify our beliefs in an organized and more reliable way. It could in principle be the case that a scientific investigation leads you to the conclusion that we should use other different rules, because they would be even better than what we now call science. But we would use science to get there, or whatever our CURRENT learning method is. Likewise we should change our values according to what we currently value and know.
We should design AI such that if it determines that we would consider 'personal uniqueness' extremely important if we were superintelligent, then it will strongly avoid any highly accurate simulations, even if that costs some accuracy. (Unless outweighed by the importance of the problem it's trying to solve.)
If we DON'T design AI this way, then it will do many things we wouldn't want, well beyond our current beliefs about simulations.
A great deal. I discussed this in another thread, but one of the constraints of physics tells us that the maximum computational efficiency of a system, and thus its intelligence, is inversely proportional to its size (radius/volume). So its extraordinarily unlikely, near zero probability i'd say, that you'll have some big global distributed brain with a single thread of consciousness - the speed of light just kills that. The 'entity' would need to be a community (which certainly still can be coordinated entities, but its fundamentally different than a single unified thread of thought).
Moreover, I believe the likely scenario is evolutionary:
The evolution of AGI's will follow a progression that goes from simple AGI minds (like those we have now in some robots) up to increasingly complex variants and finally up to human-equivalent and human-surpassing. But all throughout that time period there will be many individual AGI's, created by different teams, companies, and even nations, thinking in different languages, created for various purposes, and nothing like a single global AI mind. And these AGI's will be competing with both themselves and humans - economically.
I agree with most of the rest of your track of thought - we modify our beliefs and values according to our current beliefs and values. But as I said earlier, its not static. Its also not even predictable. Its not even possible, in principle, to fully predict your own future state. This to me, is perhaps the final nail in the coffin for any 'perfect' self-modifying FAI theory.
Moreover, I also find it highly unlikely that we will ever be able to create a human level AGI with any degree of pre-determined reliability about its goal system whatsoever.
I find it more likely that the AGI's we end up creating will have to learn ethics, morality, etc - their goal systems can not be hard coded, and whether they turn out friendly or not is entirely dependent on what they are taught and how they develop.
In other words, friendliness is not an inherent property of AGI designs - its not something you can design in to the algorithms itself. The algorithms for an AGI give you something like an infant brain - its just a canvas, its not even a mind yet.
On what basis will they learn? You're still starting out with an initial value system and process for changing the value system, even if the value system is empty. There is no reason to think that a given preference-modifier will match humanity's. Why will they find "Because that hurts me" to be a valid point? Why will they return kindness with kindness?
You say the goal systems can't be designed in, why not?
It may be the case that we will have a wide range of semifriendly subhuman or even near human AGI's. But when we get a superhuman AGI that is smart enough to program better AGI, why can it not do that on it's own?
I am positive that 'single entity' should not have mapped to 'big distributed global brain'.
But I also think an AIXI like algorithm would be easy to parallelize and make globally distributed, and it still maximizes a single reward function.
They will have to learn by amassing a huge amount of observations and interactions, just as human infants do, and just as general agents do in AI theory (such as AIXI).
Human brains are complex, but very little of that complexity is actually precoded in the DNA. For humans values, morals, and high level goals are all learned knowledge, and have varied tremendously over time and cultures.
Well, if you raised the AI as such, it would.
Consider that a necessary precursor of of following the strategy 'returning kindness with kindness' is understanding what kindness itself actually is. If you actually map out that word, you need a pretty large vocabulary to understand it, and eventually that vocabulary rests on grounded verbs and nouns. And to understand those, they must be grounded on a vast pyramid of statistical associations acquired from sensorimotor interaction (unsupervised learning .. aka experience). You can't program in this knowledge. There's just too much of it.
From my understanding of the brain, just about every concept has (or can potentially have) associated hidden emotional context: "rightness" and "wrongness", and those concepts: good, bad, yes, no, are some of the earliest grounded concepts, and the entire moral compass is not something you add later, but is concomitant with early development and language acquisition.
Will our AI's have to use such a system as well?
I'm not certain, but it may be such a nifty, powerful trick, that we end up using it anyway. And even if there is another way to do that is still efficient, it may be that you can't really understand human languages unless you also understand the complex web of value. If nothing else, this approach certainly gives you control over the developing AI's value system. It appears for human minds the value system is immensely complex - it is intertwined at a fundamental level with the entire knowledge base - and is inherently memetic in nature.
What is an AGI? It is a computer system (hardware), some algorithms/code (which actually is always eventually better to encode directly in hardware - 1000X performance increase), and data (learned knowledge). The mind part - all the qualities of importance, comes solely from the data.
So the 'programming' of the AI is not that distinguishable from the hardware design. I think AGI's will speed this up, but not nearly as dramatically as people here think. Remember humans don't design new computers anymore anyway. Specialized simulation software does the heavy lifting - and it is already the bottleneck. An AGI would not be better than this specialized software at its task (generalized vs specialized). It will be able to improve it some almost certainly, but only to the theoretical limits, and we are probably already close enough to them that this improvement will be minor.
AGI's will have a speedup effect on moore's law, but I wouldn't be surprised if this just ends up compensating for the increased difficulty going forward as we approach quantum limits and molecular computing.
In any case, we are simulation bound already and each new generation of processors designs (through simulation) the next. The 'FOOM' has already begun - it began decades ago.
Well I'm pretty certain that AIXI like algorithms aren't going to be directly useful - perhaps not ever, only more as a sort of endpoint on the map.
But that's beside the point.
If you actually use even a more practical form of that general model - a single distributed AI with a single reward function and decision system, I can show you how terribly that scales. Your distributed AI with a million PC's is likely to be less intelligent than a single AI running on tightly integrated workstation class machine with just say 100x the performance of one of your PC nodes. The bandwidth and the latency issues are just that extreme.
If concepts like kindness are learned with language and depend on a hidden emotional context, then where are the emotions learned?
What is the AI's motivation? This is related to the is-ought problem: no input will affect the AI's preferences unless there is something already in the AI that reacts to that input that way.
If software were doing the heavy lifting, then it would require no particular cleverness to be a microprocessor design engineer.
The algorithm plays a huge role in how powerful the intelligence will be, even if it is implemented in silicon.
People might not make most of the choices in laying out chips, but we do almost all of the algorithm creation, and that is where you get really big gains. see Deep Fritz vs. Deep Blue. Better algorithms can let you cut out a billion tests and output the right answer on the first try, or find a solution you just would not have found with your old algorithm.
Software didn't invent out of order execution. It just made sure that the design actually worked.
As for the distributed AI: I was thinking of nodes that were capable of running and evaluating whole simulations, or other large chunks of work. (Though I think superintelligence itself doesn't require more than a single PC.)
In any case, why couldn't your supercomputer foom?
I think this is an open question, but certainly one approach is to follow the brain's lead and make a system that learns its ethics and high level goals dynamically, through learning.
In that type of design, the initial motivation gets imprinting queues from the parents.
Oh of course, but I was just pointing out that after a certain amount of research work in a domain, your algorithms converge on some asymptotic limit for the hardware. There is nothing even close to unlimited gains purely in software.
And the rate of hardware improvement is limited now by speed of simulation on current hardware, and AGI can't dramatically improve that.
Yes, of course. Although as a side note we are moving away from out of order execution at this point.
Because FOOM is just exponential growth, and in that case FOOM is already under way. It could 'hyper-FOOM', but the best an AGI can do is to optimize its brain algorithms down to the asymptotic limits of its hardware, and then it has to wait with everyone else until all the complex simulations complete and the next generation of chips come out.
Now, all that being said, I do believe we will see a huge burst of rapid progress after the first human AGI is built, but not because that one AGI is going to foom by itself.
The first human-level AGI's will probably be running on GPUs or something similar, and once they are proven and have economic value, there will be this huge rush to encode those algorithms directly in to hardware and thus make them hundreds of times faster.
So I think from the first real-time human-level AGI it could go quickly to 10 to 100X AGI (in speed) in just a few years, along with lesser gains in memory and other IQ measures.