= 27d567bc2d48d9f40e9ccc20ea370b15)
Eliezer_Yudkowsky comments on Open Thread: February 2010 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (738)
A query to Unknown, with whom I have this bet going:
I recently found within myself a tiny shred of anticipation-worry about actually surviving to pay off the bet. Suppose that the rampant superintelligence proceeds to take over its future light cone but, in the process of dissembling existing humans, stores their mind-state. Some billions of years later, the superintelligence runs across an alien civilization which succeeded on their version of the Friendly AI problem and is at least somewhat "friendly" in the ordinary sense, concerned about other sentient lives; and the superintelligence ransoms us to them in exchange for some amount of negentropy which outweighs our storage costs. The humans alive at the time are restored and live on, possibly having been rescued by the alien values of the Super Happy People or some such, but at least surviving.
In this event, who wins the bet?
SIAI: Utopia or hundred times your money back!
Eliezer, would you accept a bet $100 against $10000?
On the same problem? I might attach some extra terms and conditions this time around, like "offer void (stakes will be returned) if the AI has the power and desire to use us for paperclips but our lives are ransomed by some other entity with the power to offer the AI more paperclips than it could produce by consuming us", "offer void if the explanation of the Fermi Paradox is a previously existing superintelligence which shuts down any new superintelligences produced", and "offer void if the AI consumes our physical bodies but we continue via the sort of weird anthropic scenario introduced in The Finale of the Ultimate Meta Mega Crossover." With those provisos, my probability drops off the bottom of the chart. I'm still not sure about the bet, though, because I want to keep my total of outstanding bets to something I can honor if they all simultaneously go wrong (no matter how surprising that would be to me), and this would use up $10,000 of that, even if it's on a sure thing - I might be able to get a better price on some other sure thing.
If we survive by an anthropic situation (it's hard to see how that could preserve several persons together, but just in case), then you win the bet, since that would more like a second world than a continuation of this one.
If the AI is shut down before it has had a chance to operate, the bet wouldn't have been settled yet, so you wouldn't have to pay anything.
Anyway, I'm still going to win.
You definitely win. If I say "you'll get killed doing that" and you are, I shan't expect to pay back my winnings when you're reanimated.
Perhaps you've already defined "superintelligent" as meaning "self-directed, motivated, and recursively self-improving" rather than merely "able to provide answers to general questions faster and better than human beings"... but if you haven't, it seems to me that the latter definition of "superintelligent" would have a much higher probability of you losing the bet. (For example, a Hansonian "em" running on faster hardware and perhaps a few software upgrades might fit the latter definition.)
I think I would win the bet. It wouldn't exactly be "the end of the world", but just a very strange future of the world.
Really? Huh. To me that seems both pretty world-endy and strongly against the spirit of what was implied by your original statement... would you predict this outcome? Is it something that your model allows to happen? I know it's not something I would feel compelled to make excuses for - more like "I TOLD YOU SO!"
What exactly do you think happens in the scenario described?
Ok, if you're sufficiently worried about the possibility of that outcome, I'll be happy to grant it to your side of the bet... even though at the time, it seemed to me clear that your assertion that the world would end meant that we wouldn't continue as conscious beings.
I definitely wouldn't predict that outcome. I would be very surprised, since I think the world will continue in the usual way. But is it really that likely even on your model?