= 27d567bc2d48d9f40e9ccc20ea370b15)
handoflixue comments on SotW: Be Specific - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (306)
Genie's Folly
A near omnipotent being is offering you a single wish. It is known that the Genie will attempt to implement the wish in a way the results in a net decrease of utility for the wisher, but is bound by any constraints explicitly written into the wish. Write your wish in such a way that the Genie can only implement it in such a way that you have a net increase in utility. Bonus points if you wish for something related to a current problem you are solving; ie, I wish I ran a successful startup with x following properties, which avoids y pitfalls in z ways.
Constraint: This must result in a net increase in utility for me...
I rewire your preferences. Oh, that wasn't what you meant by "utility"?
Incidentally, I find it funny (although not necessarily significant) that everyone else's instinct was to talk about the Genie in the third person, whereas Eliezer used the first person.
(Double-posted because it's a completely separate and much more frivolous comment.)
It is extremely significant. That's partly the reason why EY managed to play the AI-in-a-box game rather successfully despite the overwhelming odds.
Er.... How do you know? I thought he hadn't disclosed anything about how he did that.
He mentioned on some mailing list that he had to think like an AI desperately trying to get out. It makes a world of difference in how you approach the situation if it is your life that is actually on the line.
The role of "the Genie" here and "the AI" in the Boxed AI game have certain obvious similarities.
It seems reasonable to assume that a willingness to adopt the former correlates with a willingness to adopt the latter.
It seems reasonable to assume that a willingness to adopt the role of "the AI" in the Boxed AI game is necessary (though not sufficient) in order to win that game.
So shminux's claim seems fairly uncontroversial to me.
Do you dispute it, or are you merely making a claim about the impossibility of knowledge?
The latter.
You could try: "Constraint: The value that my current utility function would assign to the universe after this wish is implemented must be higher than the value my current utility function would assign to the universe that would have existed had you not implemented this wish."
... Which probably causes the Genie to, at best, throw an undefined-error, since human beings don't have well-defined utility functions. Since it's malicious, it will probably search through all of your desires, pick one of them at random to count as "my utility function", and then reinterpret the body of the wish to maximise that one thing at the expense of all others.
It's malicious and omnipotent. It'll do far worse than that. It'll scan your preferences until it finds a contradiction. Once you have a contradiction you can derive absolutely anything. It would then proceed to calculate your Coherent Extrapolated Volition and minimise it. It may not be obliged to figure out what you actually want but it can certainly do so for the purpose of being spiteful!
I think that is the first time I've ever seen anyone accurately describe the worst thing that could possibly happen.
Or alter my preferences so I antiprefer whatever it is able to produce the most of. Plus altering my brain such that my disutility and dishedonism are linear with that thing. Getting the attention of a Crapsack God sucks.
Those are actually subsumed under "mimimize CEV<TheOtherDave>". In the same way that maximising our CEV will not involve modifying our preferences drastically (unless it turns out we are into that sort of thing after all), minimising CEV would, if that turns out to be the worst way to @#$@ with us.
Can't argue with that.
What happens if you ask it to maximize your CEV, though?
Lemme remember, the idea with CEV was what you'd desired if you thought faster and more reliably. Okay I ponder what will happen to you if your mind was BusyBeaver(10) times faster (way scarier number than 3^^^^3), without your body working any faster. 1 second passes.
It'll fuck with you. Because that is what it does. It has plenty of scope to do so because CEV is not fully defined as of now. I'm not sure precisely how it would go about doing so. I just assume it does in some way I haven't thought of yet.
The meaning it attributes to CEV when it wants to exploit it to make things terrible is very different to the meaning it attributes to CEV when we try to use it to force it to understand us. It's almost as bad as some humans in that regard!
The understatement of the year. CEV is vaguest crap ever with lowest hope of becoming less vague.
That's a rather significant claim.
It's very uncommon to see crap this vague in development for such a long time by such a clever person, without it becoming less vague.