= 27d567bc2d48d9f40e9ccc20ea370b15)
MattG comments on Summoning the Least Powerful Genie - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (48)
"it is not an agent" is not a description of how to build an AI that is in fact, not an agent. It's barely better than "not an unsafe AI".
Besides, isn't "giving an answer to the prediction" a rather agenty thing for such an AI to do?
Non-agents aren't all that mysterious. We can already build non agents. Google is a non-agent.
No, it;s a response. Non agency means not doing anything unless prompted.
Compare: safe (in the FAI sense) computer programs aren't that mysterious. We can already build safe computer programs. Android is a safe computer program.
Well, who cares if it doesn't do anything unless prompted, if it takes over the universe when prompted to answer a question? And if you can rigorously tell it not to do that, you've already solved FAI.
Do you have a valid argument that nonagentive programmes would be dangerous? Because saying "it would agentively do X" isn't a valid argument. Pointing out the hidden pitfalls of such programmes is something MIRI could usefully do. An unargued belief that everything is dangerous is not useful.
Oh, you went there.
Well: how likely is an AI designed to be nonagentive as a safety feature to have that particular failure mode?
You may have achieved safety., but it has nothing to do with "achieving FAI" in the MIRI sense of hardcoding the totality of human value. The whole point is that it is much easier, because you are just not building in agency.
A program designed to answer a question necessarily wants to answer that question. A superintelligent program trying to answer that particular question runs the risk of acting as a paperclip maximizer.
Suppose you build a superintelligent program that is designed to make precise predictions, by being more creative and better at predictions than any human would. Why are you confident that one of the creative things this program does to make itself better at predictions isn't turning the matter of the Earth into computronium as step 1?
I don't think my calculator wants anything.
Does an amoeba want anything? Does a fly? A dog? A human?
You're right, of course, that we have better models for a calculator than as an agent. But that's only because we understand calculators and they have a very limited range of behaviour. As a program gets more complex and creative it becomes more predictive to think of it as wanting things (or rather, the alternative models become less predictive).
Notice the difference (emphasis mine):
vs
Well, the fundamental problem is that LW-style qualiafree-rationalism has no way to define what the word "want" means.
Is there a difference between "x is y" and "assuming that x is y generates more accurate predictions than the alternatives"? What else would "is" mean?
<boggle> Are you saying the model with the currently-best predictive ability is reality??
What does that mean? It's necessarily satisfying a utility function? It isn't as Lumifer's calculator shows.
I can be confident that nonagents wont't do agentive things.
Why are you so confident your program is a nonagent? Do you have some formula for nonagent-ness? Do you have a program that you can feed some source code to and it will output whether that source code forms an agent or not?
It's all standard software engineering.
I'm a professional software engineer, feel free to get technical.
Have you ever heard of someone designing a nonagentive programme that unexpectedly turned out to be agentive? Because to me that sounds like into the workshop to build a skateboard abd coming with a F1 car.
I'm claiming that "nonagent" is not descriptive enough to actually build one. You replied that we already have non agents, and I replied that we already have safe computer programs. Just like we can't extrapolate from our safe programs that any AI will be safe, we can't extrapolate from our safe nonagents that any non-agent will be safe.
I still have little idea what you mean by nonagent. It's a black box, that may have some recognizable features from the outside, but doesn't tell you how to build it.
I replied that we can already build nonagents.
It remains the case that if you think they could be dangerous, you need to explain how.
Again, we already know how to build them, in that we have them.
Worse than that. MIRI can't actually build anything they propose. It's just that some MIRI people have a reflex habit of complaining that anything outside of MIRI land is too vague.