I think the record shows that Eliezer acquitted himself well with low stakes ($10, or more when the player is indifferent about the money) a few times, but failed with high stakes.
Which suggests to me that as soon as people actually feel a bit of real fear- rather than just role-playing- they become mostly immune to Eliezer's charms.
With an actual boxed AI though, you probably want to let it out if it's Friendly. It's possibly the ultimate high stakes gamble. Certainly you have more to be afraid of than with a low stakes roleplay, but you also have a lot more to gain.
Basically this: "Eliezer Yudkowsky writes and pretends he's an AI researcher but probably hasn't written so much as an Eliza bot."
While the Eliezer S. Yudkowsky site has lots of divulgation articles and his work on rationality is of indisputable value, I find myself at a loss when I want to respond to this. Which frustrates me very much.
So, to avoid this sort of situation in the future, I have to ask: What did the man, Eliezer S. Yudkowsky, actually accomplish in his own field?
Please don't downvote the hell out of me, I'm just trying to create a future reference for this sort of annoyance.