= 27d567bc2d48d9f40e9ccc20ea370b15)
drnickbone comments on Siren worlds and the perils of over-optimised search - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (411)
This also creates some interesting problems... Suppose a very powerful AI is given human liberty as a goal (or discovers that this is a goal using coherent extrapolated volition). Then it could quickly notice that its own existence is a serious threat to that goal, and promptly destroy itself!
yes, but what about other AIs that might be created, maybe without liberty as a top goal - it would need to act to prevent them from being built! It's unlikely that "destroy itself" is the best option it can find...
Except that acting to prevent other AIs from being built would also encroach on human liberty, and probably in a very major way if it was to be effective! The AI might conclude from this that liberty is a lost cause in the long run, but it is still better to have a few extra years of liberty (until the next AI gets built), rather than ending it right now (through its own powerful actions).
Other provocative questions: how much is liberty really a goal in human values (when taking the CEV for humanity as a whole, not just liberal intellectuals)? How much is it a terminal goal, rather than an instrumental goal? Concretely, would humans actually care about being ruled over by a tyrant, as long as it was a good tyrant? (Many people are attracted to the idea of an all-powerful deity for instance, and many societies have had monarchs who were worshipped as gods.) Aren't mechanisms like democracy, separation of powers etc mostly defence mechanisms against a bad tyrant? Why shouldn't a powerful "good" AI just dispense with them?
A certain impression of freedom is valued by humans, but we don't seem to want total freedom as a terminal goal.
Well of course we don't. Total freedom is an incoherent goal: the only way to ensure total future freedom of action is to make sure nothing ever happens, thus maximizing the number of available futures without ever actually choosing one.
As far as I've been able to reason out, the more realistic human conception of freedom is: "I want to avoid having other agenty things optimize me (for their preferences (unilaterally))." The last part is there because there are mixed opinions on whether you've given up your ethical freedom if an agenty thing optimizes you for your preferences (as might happen in ideal situations, such as dealing with an FAI handing out transhuman candy), or whether you've given up your ethical freedom if you bind yourself to implement someone else's preferences mixed-in with your own (for instance, by getting married).
That doesn't make sense -- why would the status quo, whatever it is, always maximize the number of available futures? Choosing a future does not restrict you, it does close some avenues but also opens other ones.
"Total freedom" is a silly concept, of course, but it's just as silly as "Total <anything>".
Total happiness seems to make more plausible sense than total freedom.
Not sure how you determine degrees of plausibility :-/
The expression "total happiness" (other than in contexts of the "it's like, dude, I was so totally chill and happy" kind) makes no more sense to me than "total freedom".
Assume B choose without coercion, but assume A always knows what B will choose and can set up various facts in the world to determine B's choice. Is B free?
I think there is insufficient information to answer the question as asked.
If I offer you the choice of a box with $5 in it, or a box with $500 000 in it, and I know that you are close enough to a rational utility-maximiser that you will take the $500 000, then I know what you will choose and I have set up various facts in the world to determine your choice. Yet it does not seem on the face of it as if you are not free.
On the other hand if you are trying to decide between being a plumber or a blogger and I use superhuman AI powers to subtly intervene in your environment to push you into one or the other without your knowledge then I have set up various facts in the world to determine your choice and it does seem like I am impinging on your freedom.
So the answer seems to depend at least on the degree of transparency between A and B in their transactions. Many other factors are almost certainly relevant, but that issue (probably among many) needs to be made clear before the question has a simple answer.
Can you cash out the difference between those two cases in sufficient detail that we can use it to safely defined what liberty means?
I said earlier in this thread that we can't do this and that it is a hard problem, but also that I think it's a sub-problem of strong AI and we won't have strong AI until long after we've solved this problem.
I know that Word of Eliezer is that disciples won't find it productive to read philosophy, but what you are talking about here has been discussed by analytic philosophers and computer scientists as "the frame problem" since the eighties and it might be worth a read for you. Fodor argued that there are a class of "informationally unencapsulated" problems where you cannot specify in advance what information is and is not relevant to solving the problem, hence really solving them as opposed to coming up with a semi-reliable heuristic is an incredibly difficult problem for AI. Defining liberty or identifying it in the wild seems like it's an informationally unencapsulated problem in that sense and hence a very hard one, but one which AI has to solve before it can tackle the problems humans tackle.
If I recall correctly Fodor argued in Modules, Frames, Fridgeons, Sleeping Dogs, and the Music of the Spheres that this problem was in fact the heart of the AI problem, but that proposition was loudly raspberried in the literature by computer scientists. You can make up your own mind about that one.
Here's a link to the Stanford Dictionary of Philosophy page on the subject.
You mean the frame problem that I talked about here? http://lesswrong.com/lw/gyt/thoughts_on_the_frame_problem_and_moral_symbol/
The issue can be talked about in terms of the frame problem, but I'm not sure it's useful. In the classical frame problem, we have a much clearer idea of what we want, the problem is specifying enough so that the AI does too (ie so that the token "loaded" corresponds to the gun being loaded). This is quite closely related to symbol grounding, in a way.
When dealing with moral problems, we have the problem that we haven't properly defined the terms to ourselves. Across the span of possible futures, the term "loaded gun" is likely much sharply defined than "living human being". And if it isn't - well, then we have even more problems if all our terms start becoming slippery, even the ones with no moral connotations.
But in any case, saying the problem is akin to the frame problem... still doesn't solve it, alas!
It depends on how general or narrow you make the problem. Compare: is classical decision theory the heart of the AI problem? If you interpret this broadly, then yes; but the link from, say, car navigation to classical decision theory is tenuous when you're working on the first problem. The same thing for the frame problem.
Note that the relevance issue has been successfully solved in any number of complex practical applications, such as the self-driving vehicles, which are able to filter out gobs of irrelevant data, or the LHC code, which filters out even more. I suspect that the Framing Problem is not some general problem that needs to be resolved for AI to work, but just one of many technical issues, just as the "computer scientists" suggest. On the other hand, it is likely to be a real problem for FAI design, where relying to heuristics providing, say, six-sigma certainty just isn't good enough.
I think that the framing problem is distinct from the problem of defining and calculating
mostly because attempting to define liberty objectively leads us to the discussion of free will, the latter being an illusion due to the human failure to introspect deep enough.
So, just checking before I answer: you're claiming that no direct, gun-to-the-head coercion is employed, but Omega can always predict your actions and responses, and sets things up to ensure you will choose a specific thing.
Are you free, or are you in some sense "serving" Omega? I answer: The latter, very, very, very definitely.
If we take it out of abstract language, real people manipulate each-other all the time, and we always condemn it as a violation of the ethical principle of free choice. Yes, sometimes there are principles higher than free choice, as with a parent who might say, "Do your homework or you get no dessert" (treat that sentence as a metasyntactic variable for whatever you think is appropriate parenting), but we still prefer, all else equal, that our choices and those of others be less manipulated rather than more.
Just because fraud and direct coercion are the usual standards for proving a violation of free choice in a court of law, for instance in order to invalidate a legal contract, does not mean that these are the all-and-all of the underlying ethics of free choice.
Then if Omega is superintelligent, it has a problem: every single decision it makes increases or decreases the probability of someone answering something or other, possibly by a large amount. It seems Omega cannot avoid being coercive, just because it's so knowledgeable.
We don't quite know that, and there's also the matter of whether Omega is deliberately optimizing those people or they're just reacting to Omega's optimizing the inanimate world (which I would judge to be acceptable and, yes, unavoidable).
It is an interesting question, though, and illustrates the challenges with "liberty" as a concept in these circumstances.
I think Asimov did this first with his Multivac stories, although rather than promptly destroy itself Multivac executed a long-term plan to phase itself out.