It also helps to be wrong, if we are to assume that there exist false arguments that Holden would buy.
You know what would work to instantly change my opinion from 'dilettantes technobabbling' to 'geniuses talking of stuff i dont always understand'? If it is shown that AIXI is going to go evil, using math.
Quick googling finds this:
http://www.mail-archive.com/agi@v2.listbox.com/msg00749.html
Eliezer had 9 years to prove using math that AIXI is going to do something evil. Or to define something provably evil that is similar to AIXI (wouldn't raise the existential risk if AIXI is this evil, would it? ) I would accept it even if it uses some Solomonoff induction oracle.
My guess of what Eliezer had in mind for (2) is that if you control it by hooking up a reward button to it, then AIXI approximates an Outcome Pump and this is a Bad Thing.
But if that's the problem, it also illustrates why a formal proof of unfriendliness is a rather tall order. It's easy to formally specify what AIXI, or the Outcome Pump, will do. But in order prove that that's not what we want, we also need a formal specification of what we want. Which is the fundamental problem of friendliness theory.
I was wondering - what fraction of people here agree with Holden's advice regarding donations, and his arguments? What fraction assumes there is a good chance he is essentially correct? What fraction finds it necessary to determine whenever Holden is essentially correct in his assessment, before working on counter argumentation, acknowledging that such investigation should be able to result in dissolution or suspension of SI?
It would seem to me, from the response, that the chosen course of action is to try to improve the presentation of the argument, rather than to try to verify truth values of the assertions (with the non-negligible likelihood of assertions being found false instead). This strikes me as very odd stance.
Ultimately: why SI seems certain that it has badly presented some valid reasoning, rather than tried to present some invalid reasoning?
edit: I am interested in knowing why people agree/disagree with Holden, and what likehood they give to him being essentially correct, rather than a number or a ratio (that would be subject to selection bias).