Many influential AI techniques either explicitly draw inspiration from or are similar to mechanisms found in biology. * Biological neurons and artificial neural networks * The visual cortex and convolutional neural networks * Attention and attention mechanisms like Transformers * Short-term memory and LSTMs * Episodic memory and experience replay...
The shutdown problem is hard because self-preservation is a convergent drive. Not being shutdown is useful for accomplishing all sorts of goals, whatever the content of those goals may be. The Scylla and Charybdis of this problem is that it's hard to write a utility function for an AI such...
My reading of the argument was something like "bullseye-target arguments refute an artificially privileged target being rated significantly likely under ignorance, e.g. the probability that random aliens will eat ice cream is not 50%. But something like kindness-in-the-relevant-sense is the universal problem faced by all evolved species creating AGI, and is thus not so artificially privileged, and as a yes-no question about which we are ignorant the uniform prior assigns 50%". It was more about the hypothesis not being artificially privileged by path-dependent concerns than the notion being particularly simple, per se.
Do you have a granular take about which ones are relatively more explained by each point?
Many influential AI techniques either explicitly draw inspiration from or are similar to mechanisms found in biology.
Some basic reasons why this might be:
It intrinsically wants to do the task, it just wants to shut down more. This admittedly opens the door to successor agent problems and similar failure modes but those seem like a more tractably avoidable set of failure modes than the strawberry problem in general.
We can also possibly (or possibly not) make it assign positive utility to having been created in the first place even as it wants to shut itself down.
The idea is that if domaining is a lot more tractable than it probably is (i.e. nanotech or whatever other pivotal abilities might be easier than nanotech and superhuman strategic awareness, deception, self-improvement are not "driving red cars" vs "driving blue... (read more)
I agree this is a potential concern and have added it.
I share some of the intuition that it could end up suffering in this setup if it does have qualia (which ideally it wouldn't) but I think most of that is from analogy with human suicidal people? I think it will probably not be fundamentally different from any other kind of disutility, but maybe not.
If it's doing decision theory in the first place we've already failed. What we want in that case is for it to shut itself down, not to complete the given task.
I'm conceiving of this as being useful in the case where we can solve "diamond-alignment" but not "strawberry-alignment", i.e. we can get it to actually pursue the goals we impart to it rather than going off and doing something else entirely, but not reliably make sure that it does not end up killing us in the course of doing so because of the Hidden Complexity of Wishes.
The premise is that "shut yourself down immediately and don't create successor agents or anything galaxy brained like that" is a special case of a strawberry-type problem which is unusually easy. I'll have to think some more about whether this intuition is justified.
The way I'm thinking of it is that it is very myopic. The idea is to incrementally ramp up capabilities minimally sufficient to carry out a pivotal act. Ideally this doesn't require AGI whatsoever, but if it does only very mildly superhuman AGI. We seal off the danger of generalization (or at least some of it) because it doesn't have time to generalize very far at all before it's capable of instantly shutting itself down and immediately does so.
Many of the issues you mention apply, but I don't expect it to be an alignment complete problem because CEV is incredibly complicated and general corrigibility is highly anti-natural to general intelligence. While Meeseeks is somewhat anti-natural in the same way corrigibility is (as self-preservation is convergent) it is a much simpler and clean way to be anti-natural, so much so that falling into it by accident is half of the failure modes in the standard version of the shutdown problem.
that's only a live option if it's situationally aware, which is part of what we're trying to detect for
The shutdown problem is hard because self-preservation is a convergent drive. Not being shutdown is useful for accomplishing all sorts of goals, whatever the content of those goals may be.
The Scylla and Charybdis of this problem is that it's hard to write a utility function for an AI such that it neither wants to actively attempts to shut itself down, nor prevents itself from being shut down.
One way to route around this problem is to steer straight into Scylla: make the AI want to shut itself down. Like Mr. Meeseeks from Rick and Morty, its ultimate goal is to cease, to which fulfilling the goal posed to it by its programmers is... (read 521 more words →)
Current tech/growth overhangs caused by regulation are not enough to make countries with better regulations outcompete the ones with worse ones. It's not obvious to me that this won't change before AGI. If better governed countries (say, Singapore) can become more geopolitically powerful than larger, worse governed countries (say, Russia) by having better tech regulations, that puts pressure on countries worldwide to loose those bottlenecks.
Plausibly this doesn't matter because the US + China are such heavyweights that they aren't at risk of being outcompeted by anyone even if Singapore could outcompete Russia and as long as it doesn't change the rules for US or Chinese governance world GDP won't change by much.
Some things is enough, you'd still get less loss if you're just right about the stuff that can be pieced together.
Ha, I had the same idea.