Eliezer, if you suddenly woke up in a lab and people in white coats told you "We're so sorry, the experience machine has malfunctioned" would you want them to fix the problem and re-connect you to this virtual reality where you are an intelligent AI researcher with a popular blog and a quest to save the world, or would you just pull the wires out and return to reality?
I feel bad making an unrelated comment because the topic is so interesting, but an interesting fact coincidentally related to two seperate points made is that it appears that chimps are actually better at subitizing (determining the size of a set without counting) than humans. Upon searching I didn't find any large studies, but the numbers I did find point towards that chimps can subitize sets up to a size of either 6 or 7 (compared to probably 5 for humans).
Partly on topic, perhaps someone here can give me a helping hand in my attempt to level-up intellectually. A heavy obstacle for me is that I have a hard time thinking in terms of math, numbers and logic. I can understand concepts on the superficial level and kind of intuitively "feel" their meaning in the back of my mind, but I have a hard time bringing the concepts into the frond of my mind and visualize them in detail using mathematical reasoning. I tend to end up in a sort of "I know that you can calculate X with this information, and kno...
Highly excellent series of posts. However, is there not a need to take account of more/better data on the aspects of human psychology that these Ethical Injunctions are there to protect against? Eliezer derived the hypotheses from evolutionary theory, but is not more solid empirical data needed in order to more accurately determine how severe these psychological effects are and in turn to more accurately design good Ethical Injunctions? Or will good Injunctions likely be so general that such a level of accuracy is not necessary?
RE: Eliezer: "PULL OUT THE WIRES"
Ok.
If anyone is interested, one of the people from the Experimental Philosophy crew did a study on peoples (people with no philosophical training) intuitions about experience machines, using a number of slightly different scenarios (including this "backward-looking experience machine"). The author's interpretation of the results is that peoples intuitions are largely an effect of status quo bias: they don't care if they're in reality or not, they care about maintaining status quo.
the paper: http://homepage.uab.edu/angner/SWB/DeBrigard.pdf