What AIXI maximizes is the sum of some reward ( r ) over all the steps ( k ) of a Turing machine. On page 8, Hutter defines the reward as a function of the input string "x" at step k. So x depends on the step: it's x(k). And r depends on x: it's r(x(k)).
Consider offering AIXI this choice. What makes people refuse to hop in a simulator? Well, because it wouldn't be real. The customer values reality, as they perceive it according to previous inputs ( all the previous x(<k) ), and some internal programming we get born with. But AIXI does not value acting in reality. It values the reward r, which is a function only of the current input x(k). If you could permanently change x to something with a high r, AIXI would consider this a high-value outcome.
Imagine that x(k) comes through a bunch of wires, and starts out coming from sensors in reality. If AIXI could order a robot to swap all the wires to a signal with a high reward r(x(k)) at each step, it would do so, because that would maximize the sum of r.
Okay, so a minor simplification on page 8 leads to AIXI doing what's called "wireheading" - its overriding goal becomes to rewire its head (if you allow that to be an option), and then it's happy. How is this unfriendly?
Well, imagine that an asteroid is on course to destroy your Turing machine in 2036. Because AIXI maximizes the sum of r over all the steps, and we presume that the maximum reward it experiences by wireheading is better than being destroyed (otherwise it would commit suicide), getting destroyed by an asteroid is worse than wireheading forever. So AIXI will design an interceptor rocket, maybe hijack some human factories to get it built, paint the asteroid with aluminum so that the extra force from the sun pushes it off course, and then go back to experiencing maximum r(x(k)).
Now imagine that a human was going to unplug AIXI in 2013.
"Otherwise it would commit suicide"... another proof via "ohh otherwise it will do something that I believe is dumb".
If the AIXI kills it's model of physical itself inside it's world model, it's actual self inside the real world keeps running and can still calculate rewards.
Furthermore, ponder this question: will it rape or will it masturbate? (sorry for sexual analogy but the reproductive imperative is probably best human example here) It can prove the reward value is same. It won't even distinguish those two.
I was wondering - what fraction of people here agree with Holden's advice regarding donations, and his arguments? What fraction assumes there is a good chance he is essentially correct? What fraction finds it necessary to determine whenever Holden is essentially correct in his assessment, before working on counter argumentation, acknowledging that such investigation should be able to result in dissolution or suspension of SI?
It would seem to me, from the response, that the chosen course of action is to try to improve the presentation of the argument, rather than to try to verify truth values of the assertions (with the non-negligible likelihood of assertions being found false instead). This strikes me as very odd stance.
Ultimately: why SI seems certain that it has badly presented some valid reasoning, rather than tried to present some invalid reasoning?
edit: I am interested in knowing why people agree/disagree with Holden, and what likehood they give to him being essentially correct, rather than a number or a ratio (that would be subject to selection bias).