To clarify, I did not mean having the data to build a neuron-by-neuron model of the brain. I meant actually understanding the underlying algorithms those slabs of neural tissue are implementing. Think less understanding the exact structure of a bird's wing, and more understanding the concept of lift.

I think, with that level of understanding, the odds that a smart engineer (even if it's not me) couldn't find something to improve seem low.

I agree that I might not need to be able to build a human brain in software to be able to build something better, as with cars and legs.

And I agree that I might be able to build a brain in software without understanding how to do it, e.g., by copying an existing one as with ems.

That said, if I understand the principles underlying a brain well enough to build one in software (rather than just copying it), it still seems reasonable to believe that I can also build something better.

Sure. His moderation activities over the last year or so have been far more... sunglasses... moderate.

Why yes, I do also believe that political figures are held to ridiculous conversational standards as well. It's a miracle they deign to talk to anyone.

The quotes aren't all about AI.

I didn't say they were. I said that just because the speaker for a particular idea comes across as crazy doesn't mean the idea itself is crazy. That applies whether all of Eliezer's "crazy statements" are about AI, or whether none of them are.

Whoever knowingly chooses to save one life, when they could have saved two – to say nothing of a thousand lives, or a world – they have damned themselves as thoroughly as any murderer.

The most extreme presumptuousness about morality; insufferable moralism.

Funny, I actually agree with the top phrase. It's written in an unfortunately preachy, minister-scaring-the-congregation-by-saying-they'll-go-to-Hell style, which is guaranteed to make just about anyone get defensive and/or go "ick!" But if you accept the (very common) moral standard that if you can save a life, it's better to do it than not to do it, then the logic is inevitable that if you have the choice of saving one lives or two lives, by your own metric it's morally preferable to save two lives. If you don't accept the moral standard that it's better to save one life than zero lives, then that phrase should be just as insufferable.

Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction. After all, if it was that simple, we wouldn’t need a social process of science right?

I decided to be charitable, and went and looked up the post that this was in: it's here. As far as I can tell, Eliezer doesn't say anything that could be interpreted as "science exists because people are stupid, and I'm not stupid, therefore I don't need science". He claims that scientific procedures compensates for people being unwilling to let go of their pet theories and change their minds, and although I have no idea if this goal was in the minds of the people who came up with the scientific method, it doesn't seem to be false that it accomplishes this goal.

This is where Yudkowsky goes crazy autodidact bonkers. He thinks the social institution of science is superfluous, were everyone as smart as he. This means he can hold views contrary to scientific consensus in specialized fields where he lacks expert knowledge based on pure ratiocination.

Ok. I disagree with a large bit of the sequences on science and the nature of science. I've wrote a fair number of comments saying so. So I hope you will listen when I say that you are taking a strawman version of what Eliezer wrote on these issues, and it almost borders on something that I could only see someone thinking if they were trying to interpret Eliezer's words in the most negative fashion possible.

His solution: morality is the function that the brain of a fully informed subject computes to determine what's right. Laughable; pathologically arrogant.

You either didn't read that sequence carefully, or are intentionally misrepresenting it.

He thinks the social institution of science is superfluous, were everyone as smart as he.

Didn't read that sequence carefully either.

That simplicity in the information sense equates with parsimony is most unlikely; for one thing, simplicity is dependent on choice of language--an insight that should be almost intuitive to a rationalist.

You didn't read that sequence at all, and probably don't actually know what simplicity means in an information-theoretic sense.

I read your post on habit theory, and I liked it, but I don't think it's an answer to the question "What should I do?"

It's interesting to say that if you're an artist, you might get more practical use out of virtue theory, and if you're a politician, you might get more practical use out of consequentialism. I'm not sure who it is that faces more daily temptations to break the rules than the rest of us; bankers, I suppose, and maybe certain kinds of computer security experts.

Anyway, saying that morality is a tool doesn't get you out of the original need to decide which lifestyle you want in the first place. Should I be an artist, or a politician, or a banker? Why? Eliezer's answer is that there are no shortcuts and no frills here; you check and see what your brain says about what you 'should' do, and that's all there is to it. This is not exactly a brilliant answer, but it may nevertheless be the best one out there. I've never yet heard a moral theory that made more sense than that, and believe me, I've looked.

It's reasonable to insist that people put their conclusions in easily digestible bullet points to convince you to read the rest of what they've written...but if, noting that there are no such bullet points, you make the decision not to read the body text -- you should probably refrain from commenting on the body text. A license to opt-out is not the same thing as a license to offer serious criticism. Eliezer may be wrong, but he's not stupid, and he's not crazy. If you want to offer a meaningful critique of his ideas, you'll have to read them first.

Science is built around the assumption that you’re too stupid and self-deceiving to just use Solomonoff induction.

He thinks the social institution of science is superfluous, were everyone as smart as he.

This is obviously false. Yudkowsky does not claim to be able to do Solomonoff induction in his head.

In general, when Yudkowsky addresses humanity's faults, he is including himself.

I'm sure I wouldn't have done what Romney did, and not so sure about whether I would have done what Yudkowsky did. Romney wanted to hurt people for the fun of it. Yudkowsky was trying to keep people from being hurt, regardless of whether his choice was a good one.

It seems almost unfair to criticize something as a problem of LW rationality when in your second paragraph you note that professionals do the same thing.

Ask yourself honestly whether you would ever or have ever done anything comparable to what Yudkowsky did in the Roko incident or what Romney did in the hair cutting incident.

I'm not sure. A while ago, I was involved in a situation where someone wanted to put personal information of an individual up on the internet knowing that that person had an internet stalker who had a history of being a real life stalker for others. The only reason I didn't react pretty close to how Eliezer reacted in the quoted incident is that I knew that the individual in question was not going to listen to me and would if anything have done the opposite of what I wanted. In that sort of context, Eliezer's behavior doesn't seem to be that extreme. Eliezer's remarks involve slightly more caps than I think I would use in such a circumstance, but the language isn't that different.

This does connect to another issue though- the scale in question of making heated comments on the internet as opposed to traumatic bullying, are different. The questions I ask myself for what it would take to do something similar to what Eliezer did are very different than the same questions for the Romney incident.

Your basic statement does it seem have some validity. One could argue that the Romney matter reflects the circumstances where he was at the time, and what was considered socially acceptable as forms of interaction or establishing dominance hierarchies. Through most of human history, that sort of behavior would probably be considered fairly tame. But this is a weak argument- even if it was due to the circumstances that Romney was in at the time, there's no question that those were his formative years, and thus could plausibly have had a permanent impact on his moral outlook.

You can't dismiss politics just because it kills some people's minds, when so much of the available information and examples come from politics.

The problem is that even as relevant examples come from politics, those are precisely the examples that people are least likely to agree actually demonstrate the intended point in question. For example, in this case, many people who aren't on the left will downplay the Romney bullying. Given that I'm someone who dislikes Romney (both in terms of personality and in terms of policy) and am not convinced that this is at all fair, using such a controversial example seems unwise. Even if one needs to use political examples, one can use examples from 10 or 15 or 30 years ago that are well known but have had their tribalness diminish in time. For example, in this context one could use a variety of examples connected to Richard Nixon.

Someone who can't be rational about politics simply isn't a good rationalist. You can't be a rationalist about the unimportant things and rationalist about the important ones--yet call yourself a rationalist overall.

Well, we can acknowledge that we're better at being rational in some areas than we are in others. Frankly, I wouldn't mind and for reasons essentially similar to your remark would endorse some amount of reduction of the no-politics rule here. Where that becomes a problem is when one tries to connect politics to other potentially controversial issues.

what Romney did in the hair cutting incident

What's that about? (PM me if it's still taboo.)

An AI could be an extremely powerful optimizer without having a category for "humans" that mapped to our own. "Human," the way we conceive of it, is a leaky surface generalization.

A strong paperclip maximizer would understand humans as well as it had to to contend with us in its attempts to paperclip the universe, but it wouldn't care about us. And a strong optimizer programmed to maximize the values of "humans" would also probably understand us, but if we don't program into its values an actual category that maps to our conception of humans, it could perfectly well end up applying that understanding to, for example, tiling the universe with crash test dummies.

Yes, I was wrong. But my point was correct. The 781 comments applied to the Main Post So:

1. The topic was popular, like I said.

2. The post could have been promoted!

But ask yourself, would you have been so harsh on a factual error had you agreed with the message? This is the way bias works, after all, by double standard more than outright discrimination. You could say I should have been more careful. But then, when you've learned not to expect a hearing, you're not so willing to jump the hoops. But it's your loss, if you're a rationalist and if you're losing input because dissenters find it's not worth their time.

As to LukeProg providing example demonstrating welcoming dissent: you couldn't have considered my counter-balancing evidence when you downvoted before taking the time even to explore the post to which the cited comment belongs.

To LukeProg: have I made my point about the limits of dissent at LW?

I've addressed factual inaccuracies in another comment. But as for discussing karma effects--that wasn't extraneous whining but was at the heart of the discussion. If you downvote discussion of karma--like you did--simply for mentioning it, even where relevant, then you effectively soft-censor any discussion of karma. How is that rational?

LukeProg: What do you say about the grounds on which downvotes are issued for dissenting matter. Isn't it clear that this is a bias LW doesn't want to talk about; perhaps altogether doesn't want to discuss its own biases?