= 27d567bc2d48d9f40e9ccc20ea370b15)
Alicorn comments on Normal Cryonics - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (930)
Or in "The Metamorphosis of Prime Intellect".
Prime Intellect was like this close to being Friendly.
Yep, you've got to get your AI like 99.8% right for it to go wrong that way.
And given Lawrence's 42 years of life after reverting the change why ever did he not work on getting another 0.199% right? In fact, what was Caroline thinking reverting the change before they had solid plan for post Prime-Intellect survival?
Fictional characters and their mortality fetishes. Pah.
The correct interpretation of the ending (based on the excerpt from the sequel posted and an interview localroger did with an Australian radio/podcast host) is that Caroline did not really revert the change; Prime Intellect remained in control of the universe.
http://www.kuro5hin.org/prime-intellect/mopidnf.html
"The Change" was keeping humans in a simulation of the universe (and turning the actual universe into computronium) instead of in the universe itself. So when it "reversed the Change" it was still as powerful as it was before the Change. What had happened was that Prime Intellect had been convinced that the post-Change world it created was not the best way of achieving its goals, so it set up a different universe. (I imagine that, as of chapter 8, Prime Intellect's current plan for humanity is something like Buddhist-style reincarnation - after all, its highest priority is to prevent human deaths.)
Actually, I'm more tempted to say that he was friendly, just not generally intelligent enough. Some of the humans seemed really silly, though...
I've no idea what extrapolated volition would mean in a population with that many freaks :-)
I agree. Prime Intellect is absolutely friendly in that most important sense of caring about the continued existence and well-being of humans.
It was a good story, but I'm not sure that humans would have actually behaved as in that universe. Or we only saw a small subset of that universe. For example, we saw no one make themselves exponentially smarter. No one cloned themselves. No people merged consciousnesses. No one tried to convince Prime Intellect to reactivate the aliens inside of a zoo that allowed them to exist and for humanity to interact with them, without the danger of the aliens gaining control of Technology.
If I could choose between waiting around for Eliezer to make Friendly AI (or fail), I would choose the universe of Prime Intellect in a heartbeat. I don't see why Fun Theory doesn't apply there.