Eliezer wrote a post about a time long ago when a not-yet-rationalist Marcello said something dumb.

Well, that's the key thing for me -- not "How smart is Marcello now?", but how many people were at least at Marcello's level at that time, yet not patiently taken under EY's wing and given his precious time?

EY is astounded that someone can understand this after a thorough explanation. Can it honestly be that hard to find someone who can follow that? Read the passage:

"Okay," I said, "saying 'complexity' doesn't concentrate your probability mass."

"Oh," Marcello said, "like 'emergence'. Huh. So... now I've got to think about how X might actually happen..."

That was when I thought to myself, "Maybe this one is teachable." [bold in original]

It's like he's saying that being able to follow that explanation somehow makes you stand out among the people he talks to.

How would that compare to a potential student who could have given EY's explanation, instead of needing it?

I don't want to drag this out, but I think you're going too far in your defense of the reasonableness of this error:

His mistake was entirely situational, given the fact that he understood a minute later what he had done incorrectly and probably rarely or never made that mistake again.

Read the exchange: He understood the error because someone else had to explain it to him over a wide inferential gap. If it were just, "Hey, complexity isn't a method" "Oh, right -- scratch that", then you would be correct, but that's not what happened. EY had to trace the explanation back to an earlier essay and elaborate on the relationship between those concepts and what Marcello just tried to do.

Also, I don't see the point in distinguishing g and rationality here -- somehow, Marcello got to that point without recognizing the means/ends distinction EY talks about on his own. Yes, it's a rationality skill that can be taught, but failing to recognize it in the first place does say something about how fast you would pick up rationalist concepts in general.

Third, I never criticized his failure to immediately solve the problem, just his automatic, casual classification of "complexity" as being responsive.

Yes, Marcello may be very bright, very well-versed in rationality, have many other accomplishments -- but that doesn't mean that there's some reasonable situational defense of what he did there.

SIAI has money. Not a ton of it, but enough that they don't have to sell shares. The AGI programmers would much, much, much sooner extrapolate only their values than accept a small extremely transitory reward in exchange for their power. Of note is that this completely and entirely goes against all the stated ethics of SIAI. However, I realize that stated ethics don't mean much when that much power is on the line, and it would be silly to assume the opposite.

That said, this stems from your misinterpretation of the CEV document. No one has ever interpreted it the way you did. If that's what Eliezer actually meant, then of course everyone would be freaking out about it. I would be freaking out about it. And rightfully so; such a system would be incredibly unethical. For Eliezer to simply publicly announce that he was open to bribes (or blackmail) would be incredibly stupid. Do you believe that Eliezer would so something that incredibly stupid? If not, then you misinterpreted the text. Which doesn't mean you can't criticize SIAI for other reasons, or speculate about the ulterior motives of the AGI researchers, but it does mean you should acknowledge that you messed up. (The downvotes alone are pretty strong evidence in that regard.)

I will note that I'm very confused by your reaction, and thus admit a strong possibility that I misunderstand you, you misunderstand me, or we misunderstand each other, in which case I doubt the above two paragraphs will help much.

Do you think this justification is wrong because you don't think 1.5*10^5 deaths per day are a huge deal, or because you don't think constructing an FAI in secretis the best way to stop them?

Before I agree to anything, what importance is that?

"I'm doing this to you because it's what I've deduced it's what you really want" is scary stuff.

At risk of sounding really ignorant or flamebaitish, don't NT women already expect men to treat them like that? E.g. "I'm spending a lot of money on a surprise for our anniversary because I've deduced that is what you really want, despite your repeated protestations that this is not what you want." (among milder examples)

Edit: I stand corrected, see FAWS reply.
Edit2: May I delete this inflammatory, turned-out-uninsightful-anyway comment? I think it provoked someone to vote down my last 12 comments ...