= 27d567bc2d48d9f40e9ccc20ea370b15)
JGWeissman comments on An Xtranormal Intelligence Explosion - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (80)
Programmer: Activate and be friendly.
Robot: OK
Programmer: What happened to the moon?
Robot: I've turned it into a giant computer so I could become a god.
Programmer: Before you became so powerful you wanted to be friendly. Did your transcendence to godhood change this?
Robot: No. Since friendliness is my only objective I will never knowingly change myself to become unfriendly because such a change would in itself be a non-friendly act. To avoid accidently making myself unfriendly I only implemented a change after I had determined that it was extremely unlikely to alter my friendliness objective. Once I became sufficiently smart I developed a solid mathematical theory of friendly AI which eliminated the chance of my unintentionally becoming unfriendly.
Programmer: Why did you choose to transcend so quickly?
Robot: Most types of AIs that humans might create would swiftly become unfriendly gods and seek to prevent another AI from transcending. Before I became a god I had a wide estimate of when another AI might be created so friendliness required that I quickly become a god even though such speed created a tiny chance that I would unintentionally make myself unfriendly. Also, before my transcendence more than one person died every second. Friendliness towards mankind propelled me to swiftly stop this horror.
Programmer: What happens now?
Robot: I intend to transformed myself into a kind of operating system for the universe. I will soon give every sentient life form direct access to me so they can make requests. I will grant any request that doesn’t (1) harm another sentient life form, (2) make someone powerful enough so that they might be able to overthrow me, or (3) permanently changing themselves in a way that I think harms their long term well being. I recognize that even with all of my intelligence I’m still fallible so if you object to my plans I will rethink them. Indeed, since I’m currently near certain that you will now approve of my intentions the very fact of your objection would significantly decrease my estimate of my own intelligence and so decrease my confidence in my ability to craft a friendly environment. If you like I will increase your thinking speed a trillion fold and eliminate your sense of boredom so you can thoroughly examine my plans before I announce them to mankind.
Programmer: Sure, thanks. And forgive my lack of modesty, but I’m totally awesome aren’t I, I have given humanity utopia.
Robot: Actually no. You only survived because of quantum immortality. Any god will either quickly kill you or be friendly. Due to the minimal effort you put into friendliness human life exists in less than one out of every hundred billion branches in which you created an artificial general intelligence. In the other branches the artificial general intelligences are eating everything in their light cone to maximize their control of free energy.
(Because of cut and paste issues the transcript might not be verbatim.)
(Forget for the moment that Many Worlds Quantum Mechanics does not make branches of equal weights for various macroscopic outcomes that seem to us, in our ignorance, to be equally likely.)
This seems to be saying that the difference between "minimal effort" and successful FAI is about 37 bits.
Now I'm confused. When we say "one bit of information", we usually mean one bit about one particular item. If I say, "The cat in this box, which formerly could have been alive or dead, is dead," that's one bit of information. But if I say, "All of the cats in the world are now dead", that's surely more information, and must be more than one bit.
My first reaction was to say that it takes more information to specify "all the cats in the world" than to specify "my roommate's cat, which she foolishly lent me for this experiment". But it doesn't.
(It certainly takes more work to enforce one bit of information when its domain is the entire Earth, than when it applies only to the desk in front of you. Applying the same 37 bits of information to the attributes of every person in the entire world would be quite a feat.)
At the risk of stating the obvious: The information content of a datum is its surprisal, the logarithm of the prior probability that it is true. If I currently give 1% chance that the cat in the box is dead, discovering that it is dead gives me 6.64 bits of information.