= 27d567bc2d48d9f40e9ccc20ea370b15)
Stuart_Armstrong comments on Tools want to become agents - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (81)
That's a complicated and interesting question, that quite a few smart people have been thinking about. Fortunately, I don't need to solve it to get the point above.
How do you decide at what point your grasp of a hypothetical system is sufficient, and the probability that it will be build large and robust enough, for it to make sense to start thinking about hypothetical failure modes?
? Explain. I can certainly come up with two hypothetical AI designs, call one a tool and the other an agent (and expect that almost everyone would agree with this, because tool vs agent is clearer at the extremities than in the middle), set up a toy situation, and note that the tools top plan is to make itself into the agent design. The "tool wants to be agent" is certainly true, in this toy setup.
The real question is how much this toy example generalises to real-world scenarios, which is a much longer project. Daniel Dewey has been doing some general work in that area.
My perception, possibly misperception, is that you are too focused on vague hypotheticals. I believe that it is not unlikely that future tool AI will be based on, or be inspired by (at least partly), previous generations of tool AI that did not turn themselves into agent AIs. I further believe that, instead of speculating about specific failure modes, it would be fruitful to research whether we should expect some sort of black swan event in the development of these systems.
I think the idea around here is to expect a strong discontinuity and almost completely dismiss current narrow AI systems. But this seems like black-and-white thinking to me. I don't think that current narrow AI systems are very similar to your hypothetical superintelligent tools. But I also don't think that it is warranted to dismiss the possibility that we will arrive at those superintelligent tools by incremental improvements of our current systems.
What I am trying to communicate is that it seems much more important to me to technically define at what point you believe tools to turn into agents, rather than using it as a premise for speculative scenarios.
Another point I would like to make is that researching how to create the kind of tool AI you have in mind, and speculating about its failure modes, are completely intervened problems. It seems futile to come up with vague scenarios of how these completely undefined systems might fail, and to expect to gain valuable insights from these speculations.
I also think that it would make sense to talk about this with experts outside of your social circles. Do they believe that your speculations are worthwhile at this point in time? If not, why not?
Just because I haven't posted on this, doesn't mean I haven't been working on it :-) but the work is not yet ready.
That's exactly what the plan is now: I think I have enough technical results that I can start talking to the AI and AGI designers.
I'm curious - who are the AI and AGI designers- seeing one hasn't been publicly built yet. Or is this other researchers in the AGI field. If you are looking for feedback from a technical though not academic, I'd be very interested in assisting.
There are a half-dozen AGI projects with working implementations. There are multiple annual conferences where people working on AGI share their results. There's literature on the subject going back decades, really to the birth of AI in the 50's and 60's. The term AGI itself was coined by people working in this field to describe what they are building. Maybe you mean something different than AGI when say "one hasn't been publicly built yet" ?
There seems to be some serious miscommunication going on here. By "AGI", do you mean a being capable of a wide variety of cognitive tasks, including passing the Turing Test? By "AGI project", do you mean an actual AGI, and not just a project with AGI as its goal? By "working implementation", do you mean actually achieving AGI, or just achieving some milestone on the way?
I meant Artificial General Intelligence as that term has been first coined and used in the AI community: the ability to adapt to any new environment or task.
Google's machine learning algorithms can not just correctly classify videos of cats, but can innovate the concept of a cat given a library of images extracted from video content, and no prior knowledge or supervisory feedback.
Roomba interacts with its environment to build a virtual model of my apartment, and uses that acquired knowledge to efficiently vacuum my floors while improvising in the face of unexpected obstacles like a 8mo baby or my cat.
These are both prime examples of applied AI in the marketplace today. But ask Google's neural net to vacuum my floor, or a Roomba to point out videos of cats on the internet and ... well the hypothetical doesn't even make sense -- there is an inferential gap here that can't be crossed as the software is incapable of adapting itself.
A software program which can make changes to its own source code -- either by introspection or random mutation -- can eventually adapt to whatever new environment or goal is presented to it (so long as the search process doesn't get stuck on local maxima, but that's a software engineering problem). Such software is Artificial General Intelligence, AGI.
OpenCog right now has a rather advanced evolutionary search over program space at its core. On youtube you can find some cool videos of OpenCog agents learning and accomplishing arbitrary goals in unstructured virtual environments. Because of the unconstrained evolutionary search over program space, this is technically an AGI. You could put it in any environment with any effectors and any goal and eventually it would figure out both how that goal maps to the environment and how to accomplish it. CogPrime, the theoretical architecture OpenCog is moving towards, is "merely" an addition of many, many other special-purpose memory and heuristic components which both speed the process along and make the agent's thinking process more human-like.
Notice there is nothing in here about the Turing test, nor should there be. Nor is there any requirement that the intelligence be human-level in any way, just that it could be given enough processing power and time. Such intelligences already exist.
"Pass the Turing Test" is a goal, and is therefore a subset of GI. The Wikipedia article says "Artificial general intelligence (AGI) is the intelligence of a (hypothetical) machine that could successfully perform any intellectual task that a human being can."
Your claim that OpenCog can "eventually" accomplish any task is unsupported, is not something that has been "implemented", and is not what is generally understood as what AGI refers to.
Can you list the 6 working AGI projects - I'd be interested but I suspect we are talking about different things.
OpenCog, NARS, LIDA, Soar, ACT-R, MicroPsi. More:
http://wiki.opencog.org/w/AGI_Projects http://bicasociety.org/cogarch/architectures.htm
Not sure yet - taking advice. The AI people are narrow AI developers, and the AGI people are those that are actually planning to build an AGI (eg Ben Goertzl).
For a very different perspective from both narrow AI and to a lesser extent Goertzel*, you might want to contact Pat Langley. He is taking a Good Old-Fashioned approach to Artificial General Intelligence:
http://www.isle.org/~langley/
His competing AGI conference series:
http://www.cogsys.org/
EDIT: Not that I think you shouldn't be talking to Goertzel! In fact I think his CogPrime architecture is the only fully fleshed out AGI design which as specified could reach and surpass human intelligence, and the GOLUM meta-AGI architecture is the only FAI design I know of. My only critique is that certain aspects of it are cutting corners, e.g. the rule-based PLN probabilistic reasoning engine vs an actual Bayes net updating engine a la Pearl et al.
Thanks!
It would be helpful if you spelt out your toy situation, since my intuition are currently running in the opposite direction.
It would, however, be interesting to. This discussion has come around before. What I said there:
A high-frequency trading system seems no more complex or agenty to me than rigging a shotgun to shoot at a door when someone opens the door from the outside. Am I wrong about this?
To be clear, what I think I know about high-frequency trading systems is that through technology they are able to front run certain orders they see to other exchanges when these orders are being sent to multiple exchanges in a non-simultaneous way. The thing that makes them unfriendly is that they are designed by people who understand order dynamics at the microsecond level to exploit people who trade lots of stock but don't understand the technicalities of order dynamics. That market makers are allowed to profit by selling information flow to high-frequency traders that, on examination, allows them to subvert the stated goals of a "fair" market is all part of the unfriendliness.
But high-frequency programs execute pretty simple instructions quite repeatably, they are not adaptive in a general sense or even particularly complex, they are mostly just fast.
Mmm ... I think we're arguing definitions of ill-defined categories at this point. Sort of "it's not an AI if I understand it." I was using it as an example of a "daemon" in the computing sense, a tool trusted to run without further human intervention - not something agenty.
Intentionality meaning " "the power of minds to be about, to represent, or to stand for, things, properties and states of affairs", ...or intentionally meaning purpose?
And also: Question-answerer->tool->agent is a natural progression just in process automation. (And this is why they're called "daemons".)
I'm suspecting "tool" versus "agent" is a magical category whose use is really talking about the person using it.
Thanks, that's another good point!
I think the concepts are clear at the extremes, but they tend to get muddled in the middle.
Do you believe that humans are agents? If so, what would you have to do to a human brain in order to turn a human into the other extreme, a clear tool?
I could ask the same about C. elegans. If C. elegans is not an agent, why not? If it is, then what would have to change in order for it to become a tool?
And if these distinctions don't make sense for humans or C. elegans, then why do you expect them to make sense for future AI systems?
Both your examples are agents currently. A calculator is a tool.
Anyway, I've still got a lot more work to do before I seriously discuss this issue.
I'd be especially interested in edge cases. Is e.g. Google's driverless car closer to being an agent than a calculator? If that is the case, then if intelligence is something that is independent of goals and agency, would adding a "general intelligence module" make Google's driverless dangerous? Would it make your calculator dangerous? If so, why would it suddenly care to e.g. take over the world if intelligence is indeed independent of goals and agency?
A driverless car is firmly is on the agent side of the fence, by my defintions. Feel free to state your own, anybody.
A cat's an agent. It has goals it works towards. I've seen cats manifest creativity that surprised me.
Why is that surprising? Does anyone think that "agent" implies human level intelligence?