= 27d567bc2d48d9f40e9ccc20ea370b15)
wedrifid comments on SotW: Be Specific - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (306)
You could try: "Constraint: The value that my current utility function would assign to the universe after this wish is implemented must be higher than the value my current utility function would assign to the universe that would have existed had you not implemented this wish."
... Which probably causes the Genie to, at best, throw an undefined-error, since human beings don't have well-defined utility functions. Since it's malicious, it will probably search through all of your desires, pick one of them at random to count as "my utility function", and then reinterpret the body of the wish to maximise that one thing at the expense of all others.
It's malicious and omnipotent. It'll do far worse than that. It'll scan your preferences until it finds a contradiction. Once you have a contradiction you can derive absolutely anything. It would then proceed to calculate your Coherent Extrapolated Volition and minimise it. It may not be obliged to figure out what you actually want but it can certainly do so for the purpose of being spiteful!
I think that is the first time I've ever seen anyone accurately describe the worst thing that could possibly happen.
Or alter my preferences so I antiprefer whatever it is able to produce the most of. Plus altering my brain such that my disutility and dishedonism are linear with that thing. Getting the attention of a Crapsack God sucks.
Those are actually subsumed under "mimimize CEV<TheOtherDave>". In the same way that maximising our CEV will not involve modifying our preferences drastically (unless it turns out we are into that sort of thing after all), minimising CEV would, if that turns out to be the worst way to @#$@ with us.
Can't argue with that.
What happens if you ask it to maximize your CEV, though?
Lemme remember, the idea with CEV was what you'd desired if you thought faster and more reliably. Okay I ponder what will happen to you if your mind was BusyBeaver(10) times faster (way scarier number than 3^^^^3), without your body working any faster. 1 second passes.
It'll fuck with you. Because that is what it does. It has plenty of scope to do so because CEV is not fully defined as of now. I'm not sure precisely how it would go about doing so. I just assume it does in some way I haven't thought of yet.
The meaning it attributes to CEV when it wants to exploit it to make things terrible is very different to the meaning it attributes to CEV when we try to use it to force it to understand us. It's almost as bad as some humans in that regard!
The understatement of the year. CEV is vaguest crap ever with lowest hope of becoming less vague.
That's a rather significant claim.
It's very uncommon to see crap this vague in development for such a long time by such a clever person, without it becoming less vague.
As far as I am aware this crap isn't in development. It isn't the highest research priority so the other SingInst researchers haven't been working on it much and Eliezer himself is mostly focused on writing a rationality book. Other things like decision theory are being worked on - which has involved replacing vague as crap TDT with less-vague UDT and UDT2.
I would like to see more work published on CEV. The most recent I am familiar with is this.
As I've figured out while writing the last few posts, TDT hasn't been explained well, but it is a genuinely formalizable theory. (You'll have to trust me until Part III or check the decision-theory mailing list.) But it's a different theory from ADT and UDT, and the latter ones are preferable.
You mean you have something in mind about how to handle counterfactuals over logically impossible worlds, or simply “I'm not sure it can't be done”?