All of wuwei's Comments + Replies

CEV is not preference utilitarianism, or any other first-order ethical theory. Rather, preference utilitarianism is the sort of thing that might be CEV's output.

Matt Simpson was talking about people who have in fact reflected on their values a lot. Why did you switch to talking about people who think they have reflected a lot?

What "someone actually values" or what their "terminal values" are seems to be ambiguous in this discussion. On one reading, it just means what motivates someone the most. In that case, your claims are pretty plausible.

On the other reading, which seems more relevant in this thread and the original comment, it means the terminal values someone should act on, which we might ... (read more)

I suppose I might count as someone who favors "organismal" preferences over confusing the metaphorical "preferences" of our genes with those of the individual. I think your argument against this is pretty weak.

You claim that favoring the "organismal" over the "evolutionary" fails to accurately identify our values in four cases, but I fail to see any problem with these cases.

• I find no problem with upholding the human preference for foods which taste fatty, sugary and salty. (Note that consistently applied, the "

Hi.

I've read nearly everything on less wrong but except for a couple months last summer, I generally don't comment because a) I feel I don't have time, b) my perfectionist standards make me anxious about meeting and maintaining the high standards of discussion here and c) very often someone has either already said what I would have wanted to say or I anticipate from experience that someone will very soon.

There's the consequentialist/utilitarian click, and the intelligence explosion click, and the life-is-good/death-is-bad click, and the cryonics click.

I can find a number of blog posts from you clearly laying out the arguments in favor of each of those clicks except the consequentialism/utilitarianism one.

What do you mean by "consequentialism" and "utilitarianism" and why do you think they are not just right but obviously right?

d) should be changed to the sparseness of intelligent aliens and limits to how fast even a superintelligence can extend its sphere of influence.

Interesting, what about either of the following:

A) If X should do A, then it is rational for X to do A.

B) If it is rational for X to do A, then X should do A.

I'm a moral cognitivist too but I'm becoming quite puzzled as to what truth-conditions you think "should" statements have. Maybe it would help if you said which of these you think are true statements.

1) Eliezer Yudkowsky should not kill babies.

2) Babyeating aliens should not kill babies.

3) Sharks should not kill babies.

4) Volcanoes should not kill babies.

5) Should not kill babies. (sic)

The meaning of "should not" in 2 through 5 are intended to be the same as the common usage of the words in 1.

I don't think we anticipate different experimental results.

I find that quite surprising to hear. Wouldn't disagreements about meaning generally cash out in some sort of difference in experimental results?

On your analysis of should, paperclip maximizers should not maximize paperclips. Do you think this is a more useful characterization of 'should' than one in which we should be moral and rational, etc., and paperclip maximizers should maximize paperclips?

Do you think that morality or rationality recommends placing no intrinsic weight or relevance on either a) backwards-looking considerations (e.g. having made a promise) as opposed to future consequences, or b) essentially indexical considerations (e.g. that I would be doing something wrong)?

Its painfulness.

After some medical procedure, there have been some patients for whom pain is not painful. When asked whether their pain is still there, they will report that the sensation of pain is still there just as it was before, but that they simply don't mind it anymore.

That feature of pain that their pain now lacks is what I am calling its painfulness and that is what is bad about pain.

Yes. And since being a maverick has a similar negative expectation for most working people, it seems well-placed to explain the slow spread of good ideas more generally as well.

Great post.

I agree that you identify a very good reason to take care in the use of gender-specific pronouns or anything else that is likely to create in-group, out-group effects.

I also think there probably was a fair amount of attitude polarization on the question of how acceptable it was to make the statement in question.

Under what conditions do you normally find it necessary to attempt to fully describe a goal?

Upvoted because I appreciate Alicorn's efforts and would like to hear additional rational presentations of views in the same neighborhood as her's.

I would bet I also upvoted some of the comments Alicorn is referring to as comments that perpetuate the problem.

disregard for the autonomy of people =/= thinking of someone in a way that doesn't include respect for his goals, interests, or personhood

I am reading the latter rather literally in much the same way RobinHanson seems to and as I think the author intended.

Nice. Tying the usage of words to inferences seems to be a generally useful strategy for moving semantic discussions forward.

I had negative associations attached to Roko's comment because I started imagining myself with my preferences adopting Roko's suggestions.

This sentence was meant to explain why I was momentarily off-put. I did not mean to imply that I have any ethical problems with the desires mentioned (I don't), though now that you mention it, I wouldn't be too surprised if I do retain some knee-jerk ethical intuitions against them.

Have you tried programming in a language with an interactive interpreter, extensive documentation and tutorials, and open source code?

I still have very little idea what you mean by 'objectification' and 'objectify people'.

I was momentarily off-put by Roko's comment on the desire to have sex with extremely attractive women that money and status would get. This was because of:

• the focus on sex, whereas I would desire a relationship.
• the connotation of 'attractive' which in my mind usually means physical attractiveness, whereas my preferences are dominated by other features of women.
• the modifier 'extremely' which seems to imply a large difference in utility placed on sex with extremely at

"The Fermi paradox is actually quite easily resolvable. There are zillions of aliens teeming all around us. They're just so technologically advanced that they have no trouble at all hiding all evidence of their existence from us."

I thought Chalmers is an analytic functionalist about cognition and only reserves his brand of dualism for qualia.

Yes, but not all self-help needs to involve positive affirmations.

I was going to ask whether repeating positive statements about oneself has actually been recommended on lesswrong. Then I remembered this post. Perhaps that post would have made a more suitable target than the claim that rationalists should win.

Wouldn't a rationalist looking to win simply welcome this study along with any other evidence about what does or does not work?

Unless that changes then, I wouldn't particularly recommend programming as a job. I quite like my programming job but that's because I like programming and I don't work in a dilbert cartoon.

According to an old story, a lord of ancient China once asked his physician, a member of a family of healers, which of them was the most skilled in the art.

The physician, whose reputation was such that his name became synonymous with medical science in China, replied, "My eldest brother sees the spirit of sickness and removes it before it takes shape and so his name does not get out of the house."

"My elder brother cures sickness when it is still extremely minute, so his name does not get out of the neighborhood."

"As for me, I punc

Do you program for fun?

Take the thoughts of such an one, used for many years to one tract, out of that narrow compass he has been all his life confined to, you will find him no more capable of reasoning than almost a perfect natural. Some one or two rules on which their conclusions immediately depend you will find in most men have governed all their thoughts; these, true or false, have been the maxims they have been guided by. Take these from them, and they are perfectly at a loss, their compass and polestar then are gone and their understanding is perfectly at a nonplus; and t

There is a mathematical style in which proofs are presented as strings of unmotivated tricks that miraculously do the job, but we found greater intellectual satisfaction in showing how each next step in the argument, if not actually forced, is at least something sweetly reasonable to try. Another reason for avoiding [pulling] rabbits [out of the magicians's hat] as much as possible was that we did not want to teach proofs, we wanted to teach proof design. Eventually, expelling rabbits became another joy of my professional life.

-- Edsger Dijkstra

Edit: Added context to "rabbits" in brackets.

Thanks for the explanations.

Testing shows the presence, not the absence of bugs.

-- Edsger Dijkstra

Here's one way this could be explained: Susie realizes that her name could become a cheap and effective marketing tool if she sells seashells at the seashore. Since that's something she enjoys doing anyway, she does so.

If that's how things are, I wouldn't really call this a cognitive bias.

That's a good point, but it would be more relevant if this were a policy proposal rather than an epistemic probe.

To answer your second question: No, there aren't any historical examples I am thinking of. Do you find many historical examples of existential risks?

Edit: Global nuclear warfare and biological weapons would be the best candidates I can think of.

If you decreased the intelligence of everyone to 100 IQ points or lower, I think overall quality of life would decrease but that it would also drastically decrease existential risks.

Edit: On second thought, now that I think about nuclear and biological weapons, I might want to take that back while pointing out that these large threats were predominantly created by quite intelligent, well-intentioned and rational people.

You seem to be assuming that the relation between IQ and risk must be monotonic.

I think existential risk mitigation is better pursued by helping the most intelligent and rational efforts than by trying to raise the average intelligence or rationality.

And I will suggest in turn that you are guilty of the catchy fallacy name fallacy. The giant cheesecake fallacy was originally introduced as applying to those who anthropomorphize minds in general, often slipping from capability to motivation because a given motivation is common in humans.

I'm talking about a certain class of humans and not suggesting that they are actually motivated to bring about bad effects. Rather all it takes is for there to be problems where it is significantly easier to mess things up than to get it right.

I think many of the most pressing existential risks (e.g. nanotech, biotech and AI accidents) come from the likely actions of moderately intelligent, well-intentioned, and rational humans (compared to the very low baseline). If that is right then increasing the number of such people will increase rather than decrease risk.

Increases in rationality can sometimes lead with some regularity to decreasing knowledge or utility (hopefully only temporarily and in limited domains).

I suspect you aren't sufficiently taking into account the magnitude of people's irrationality and the non-monotonicity of rationality's rewards. I agree that intelligence enhancement would have greater overall effects than rationality enhancement, but rationality's effects will be more careful and targeted -- and therefore more likely to work as existential risk mitigation.

I'm not sure intelligence enhancement alone is sufficient. It'd be better to first do rationality enhancement and then intelligence enhancement. Of course that's also much harder to implement but who said it would be easy?

It sounds like you think intelligence enhancement would result in rationality enhancement. I'm inclined to agree that there is a modest correlation but doubt that it's enough to warrant your conclusion.

"One can measure the importance of a scientific work by the number of earlier publications rendered superfluous by it."

-- David Hilbert

"Muad'Dib learned rapidly because his first training was in how to learn. And the first lesson of all was the basic trust that he could learn. It's shocking to find how many people do not believe they can learn, and how many more believe learning to be difficult. Muad'Dib knew that every experience carries its lesson."

-- Frank Herbert, Dune

"Science is what we understand well enough to explain to a computer. Art is everything else we do. ... Science advances whenever an Art becomes a Science. And the state of the Art advances too because people always leap into new territory once they have understood more about the old."

-- Donald Knuth

I like some of the imagery but I wouldn't say whatever the outcome is, it is by definition good.

To continue with the analogy, sometimes our inner book of morals really says one thing while a momentary upset prevents what is written in that book from successfully governing.

You seem to think an FAI researcher is someone who does not engage in any AGI research. That would certainly be a rather foolish researcher.

Perhaps you are being fooled by the fact that a decent FAI researcher would tend not to publicly announce any advancements in AGI research.

Would you bet on resource depletion?

Voted Down. Sorry, Roko.

I don't find Greene's arguments to be valuable or convincing. I won't defend those claims here but merely point out that this post makes it extremely inconvenient to do so properly.

I would prefer concise reconstructions of important arguments over a link to a 377 page document and some lengthy quotes, many of which simply presuppose that certain important conclusions have already been established elsewhere in the dissertation.

As an exercise for the reader demonstrating my complaint, consider what it would take to work out whether Jo... (read more)

Thanks for clarifying what factors you think are relevant. I agree that those have not been tested.

The correlations with independent ratings of attractiveness were still .44 and .39. Compared to .04 and -.06 for intelligence, that still supports the conclusion that "sheer physical attractiveness appears to be the overriding determinant of liking."

They also used various personality measures assessing such things as social skills, maturity, masculinity/femininity, introversion/extroversion and self-acceptance. They found predominantly negative correlations (from -.18 to -.08) and only two comparatively small positive correlations .14 and .03.