= 27d567bc2d48d9f40e9ccc20ea370b15)
TheAncientGeek comments on Summoning the Least Powerful Genie - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (48)
Do you have a valid argument that nonagentive programmes would be dangerous? Because saying "it would agentively do X" isn't a valid argument. Pointing out the hidden pitfalls of such programmes is something MIRI could usefully do. An unargued belief that everything is dangerous is not useful.
Oh, you went there.
Well: how likely is an AI designed to be nonagentive as a safety feature to have that particular failure mode?
You may have achieved safety., but it has nothing to do with "achieving FAI" in the MIRI sense of hardcoding the totality of human value. The whole point is that it is much easier, because you are just not building in agency.
A program designed to answer a question necessarily wants to answer that question. A superintelligent program trying to answer that particular question runs the risk of acting as a paperclip maximizer.
Suppose you build a superintelligent program that is designed to make precise predictions, by being more creative and better at predictions than any human would. Why are you confident that one of the creative things this program does to make itself better at predictions isn't turning the matter of the Earth into computronium as step 1?
I don't think my calculator wants anything.
Does an amoeba want anything? Does a fly? A dog? A human?
You're right, of course, that we have better models for a calculator than as an agent. But that's only because we understand calculators and they have a very limited range of behaviour. As a program gets more complex and creative it becomes more predictive to think of it as wanting things (or rather, the alternative models become less predictive).
Notice the difference (emphasis mine):
vs
Well, the fundamental problem is that LW-style qualiafree-rationalism has no way to define what the word "want" means.
Is there a difference between "x is y" and "assuming that x is y generates more accurate predictions than the alternatives"? What else would "is" mean?
<boggle> Are you saying the model with the currently-best predictive ability is reality??
Not quite - rather the everyday usage of "real" refers to the model with the currently-best predictive ability. http://lesswrong.com/lw/on/reductionism/ - we would all say "the aeroplane wings are real".
Errr... no? I don't think this is true. I'm guessing that you want to point out that we don't have direct access to the territory and that maps is all we have, but that's not very relevant to the original issue of replacing "I find it convenient to think of that code as wanting something" with "this code wants" and insisting that the code's desires are real.
Anthropomorphization is not the way to reality.
What does that mean? It's necessarily satisfying a utility function? It isn't as Lumifer's calculator shows.
I can be confident that nonagents wont't do agentive things.
Why are you so confident your program is a nonagent? Do you have some formula for nonagent-ness? Do you have a program that you can feed some source code to and it will output whether that source code forms an agent or not?
It's all standard software engineering.
I'm a professional software engineer, feel free to get technical.
Have you ever heard of someone designing a nonagentive programme that unexpectedly turned out to be agentive? Because to me that sounds like into the workshop to build a skateboard abd coming with a F1 car.
I've known plenty of cases where people's programs were more agentive than they expected. And we don't have a good track record on predicting which parts of what people do are hard for computers - we thought chess would be harder than computer vision, but the opposite turned out to be true.
"Doing something other than what the programmer expects" != "agentive". An optimizer picking a solution that you did not consider is not being agentive.
I haven't: have you any specific examples?
I'm claiming that "nonagent" is not descriptive enough to actually build one. You replied that we already have non agents, and I replied that we already have safe computer programs. Just like we can't extrapolate from our safe programs that any AI will be safe, we can't extrapolate from our safe nonagents that any non-agent will be safe.
I still have little idea what you mean by nonagent. It's a black box, that may have some recognizable features from the outside, but doesn't tell you how to build it.
I replied that we can already build nonagents.
It remains the case that if you think they could be dangerous, you need to explain how.
Again, we already know how to build them, in that we have them.
Worse than that. MIRI can't actually build anything they propose. It's just that some MIRI people have a reflex habit of complaining that anything outside of MIRI land is too vague.