The point is that Eliezer succeeds convincing a person who was formerly unconvinced...
Convince him about what ? If the point is to convince people about the necessity of developing a Friendly AI, then a certain amount of cheating might be justified.
Well, sure, but having now admitted that, this means that your testimony about both this and other similar scenarios constitutes very weak evidence
It's even worse than that, since I personally have never played the game against EY or anyone else, and thus I have zero credibility. I'm actually ineligible under the stated rules, because I'm fairly certain that a transhuman AI (assuming such a thing could exist, of course) could convince me of anything it wanted. I am far less certain that EY or any other human could do the same, but, sadly, this does not improve my eligibility.
...surely it would be much easier to not play the game at all, just find someone that claimed to have played it with you.
Good point. I agree.
He'd probably have to bribe me with more than I make in two years to even consider cheating in this fashion...
Fair enough, but another way to phrase this sentence is, "my testimony could be bought for a price". All that remains now is the haggling.
Lastly I think some people here may be forgetting that after the initial 2 successes by Eliezer, he repeated the AI-Box game another 3 times with raised stakes -- and his ratio of success then was 1 victory and 2 defeats.
Ah, I did not actually know that. Still, as you hint in your final sentence, it's pretty tough to know whether EY is running a Xanathos Gambit or not, due to all the secrecy.
By the way, I do not mean to imply that I'm pronouncing a moral judgement on anyone in any way -- neither on EY, nor on yourself, nor on any other AI-game players. I'm evaluating the game from a strictly rational and morality-agnostic perspective.
I recall seeing, in one of the AI-boxing discussion threads, a comment to the effect that the first step for EY to get out was to convince the other party to even play the game at all.
It has since then occurred to me that this applies to a lot of my interactions. Many people who know me IRL and know a belief of mine which they do not agree with and do not want to be convinced of often adopt the strategy of not talking with me about it at all. For me to convince one of these people of something, first I have to convince them to talk about it at all.
(Note, I don't think this is because I'm an unpleasant person to converse with. Excuses given are along the lines of "I never win an argument with you" and "you've studied it a lot more than I have, it's an unfair discussion". I don't think I'm claiming anything too outlandish here; average humans are really bad at putting rational arguments together.)
I suppose the general form is: in order to convince someone of a sufficiently alien (to them) P, first you must convince them to seriously think about P. This rule may need to be applied recursively (e.g., "seriously think about P" may require one or more LW rationality techniques).
As a practical example, my parents are very religious. I'd like to convince them to sign up for cryonics. I haven't (yet) come up with an approach that I expect to have a non-negligible chance of success. But the realization that the first goalpost along the way is to get them to seriously engage in the conversation at all simplifies the search space. (Deconversion and training in LW rationality has, of course, the best chance of success--but still a high chance of failing and I judge a failure would probably have a large negative impact on my relationship with my parents in their remaining years. That's why I'd like to convince them of just this one thing.)
I realize that this is a fairly obvious point (an application of this--raising the sanity waterline--is the point behind this entire site!), but I haven't seen this explicitly noted as being a general pattern and now that I note it, I see it everywhere--hence this post.