All of utilitymonster's Comments + Replies

I prefer this briefer formalization, since it avoids some of the vagueness of "adequate preparations" and makes premise (6) clearer.

1. At some point in the development of AI, there will be a very swift increase in the optimization power of the most powerful AI, moving from a non-dangerous level to a level of superintelligence. (Fast take-off)
2. This AI will maximize a goal function.
3. Given fast-take off and maximizing a goal function, the superintelligent AI will have a decisive advantage unless adequate controls are used.
4. Adequate controls will not

IMO, the "rapid takeoff" idea should probably be seen as a fundraising ploy. It's big, scary, and it could conceivably happen - just the kind of thing for stimulating donations.

It seems that SIAI would have more effective methods for fundraising, e.g. simply capitalizing on "Rah Singularity!". I therefore find this objection somewhat implausible.

Cool. Glad this turned out to be helpful.

A recent study by folks at the Oxford Centre for Neuroethics suggests that Greene et. al.'s results are better explained by appeal to differences in how intuitive/counterintuitive a moral judgment is, rather than differences in how utilitarian/deontological it is. I had a look at the study, and it seems reasonably legit, but I don't have any expertise in neuroscience. As I understand it, their findings suggest that the "more cognitive" part of the brain gets recruited more when making a counterintuitive moral judgment, whether utilitarian or de... (read more)

A simple explanation is that using phrases like "brain scans indicate" and including brain scan images signals scientific eliteness, and halo effect/ordinary reasoning causes them to increase their estimate of the quality of the reasoning they see.

Best rejection therapy ever.

Do you know about Giving What We Can? You may be interested in getting to know people in that community. Basically, it's a group of people that pledges to give 10% of their earnings to the most effective charities in the developing world. Feel free to PM me or reply if you want to know more.

Usually, average utilitarians are interested in maximizing the average well-being of all the people that ever exist, they are not fundamentally interested in the average well-being of the people alive at particular points of time. Since some people have already existed, this is only a technical problem for average utilitarianism (and a problem that could not even possibly affect anyone's decision).

Incidentally, not distinguishing between averages over all the people that ever exist and all the people that exist at some time leads some people to wrongly conclude that average utilitarianism favors killing off people who are happy, but less happy than average.

Gustaf Arrhenius is the main person to look at on this topic. His website is here. Check out ch. 10-11 of his dissertation Future Generations: A Challenge for Moral Theory (though he has a forthcoming book that will make that obsolete). You may find more papers on his website. Look at the papers that contain the words "impossibility theorem" in the title.

Both you and Eliezer seem to be replying to this argument:

• People only intrinsically desire pleasure.

• An FAI should maximize whatever people intrinsically desire.

• Therefore, an FAI should maximize pleasure.

I am convinced that this argument fails for the reasons you cite. But who is making that argument? Is this supposed to be the best argument for hedonistic utilitarianism?

We act not for the sake of pleasure alone. We cannot solve the Friendly AI problem just by programming an AI to maximize pleasure.

IAWYC, but would like to hear more about why you think the last sentence is supported by the previous sentence. I don't see an easy argument from "X is a terminal value for many people" to "X should be promoted by the FAI." Are you supposing a sort of idealized desire fulfilment view about value? That's fine--it's a sensible enough view. I just wouldn't have thought it so obvious that it would be a good idea to go around invisibly assuming it.

Second the need for a list of the most important problems.

How do you record your findings for future use, and how do you make sure you don't forget the important parts?

Can you explain why this analysis renders directing away from the five and toward the one permissible?

I actually don't think this is about right. Last time I asked a philosopher about this, they pointed to an article by someone (I.J. Good, I think) about how to choose the most valuable experiment (given your goals), using decision theory.

Is there data on its influence and projected influence?

Yes. They posted a bunch of self-evaluation stats. It is a start toward the information you seek.

For how many fields do you think this is possible?

Epic.

Hard to be confident about these things, but I don't see the problem with external reasons/oughts. Some people seem to have some kind of metaphysical worry...harder to reduce or something. I don't see it.

R is a categorical reason for S to do A iff R counts in favor doing A for S, and would so count for other agents in a similar situation, regardless of their preferences. If it were true that we always have reasons to benefit others, regardless of what we care about, that would be a categorical reason. I don't use the term "categorical reason" any differently than "external reason".

S categorically ought to do A just when S ought to do A, regardless of what S cares about, and it would still be true that S ought to do A in similar situat... (read more)

So are categorical reasons any worse off than categorical oughts?

I can see that you might question the usefulness of the notion of a "reason for action" as something over and above the notion of "ought", but I don't see a better case for thinking that "reason for action" is confused.

The main worry here seems to have to do with categorical reasons for action. Diagnostic question: are these more troubling/confused than categorical "ought" statements? If so, why?

Perhaps I should note that philosophers talking this way make a distinction between "motivating reasons" and &q... (read more)

I'm sort of surprised by how people are taking the notion of "reason for action". Isn't this a familiar process when making a decision?

1. For all courses of action you're thinking of taking, identify the features (consequences if you that's you think about things) that count in favor of taking that course of action and those that count against it.

2. Consider how those considerations weigh against each other. (Do the pros outweigh the cons, by how much, etc.)

3. Then choose the thing that does best in this weighing process.

The same thing can

Even if we grant that one's meta-ethical position will determine one's normative theory (which is very contentious), one would like some evidence that it would be easier to find the correct meta-ethical view than it would be to find the correct (or appropriate, or whatever) normative ethical view. Otherwise, why not just do normative ethics?

Yes, this is what I thought EY's theory was. EY? Is this your view?

On the symbolic action point, you can try making the symbolic action into a public commitment. Research suggests this will increase the strength of the effect you're talking about. Of course, this could also make you overcommit, so this strategy should be used carefully.

Especially if WBE comes late (so there is a big hardware overhang), you wouldn't need a lot of time to spend loads of subjective years designing FAI. A small lead time could be enough. Of course, you'd have to be first and have significant influence on the project.

Edited for spelling.

Don't forget about the ridiculous levels of teaching you're responsible for in that situation. Lots worse than at an elite institution.

I thought this was really, really good.

Yep, good idea.

Enjoyed most of this, some worries about how far you're getting with point 8 (on giving now rather than later).

Give now (rather than later) - I’ve seen fascinating arguments that it might be possible to do more good by investing your money in the stock market for a long period of time and then giving all the proceeds to charity later. It’s an interesting strategy but it has a number of limitations. To name just two: 1) Not contributing to charity each year prevents you from taking advantage of the best tax planning strategy available to you. That tax-

Giving What We Can does not accept donations. Just give it all to Deworm the World.

Would like to see it.

Some wisdom on warm fuzzies: http://www.pbfcomics.com/?cid=PBF162-Executive_Decision.jpg

[Not a quote, but doesn't seem suitable for a discussion article.]

My reaction is that moral philosophy just isn't science. Sure, if you're a utilitarian you can use empirical evidence to figure out what maximizes aggregate welfare, relative to your account of well-being, but you can't use science to discover that utilitarianism is true. This is because utilitarianism, like any other first-order normative theory and many meta-ethical theories, doesn't lead you to expect any experiences over any other experiences.

Thanks for writing this, Carl. I'm going to post a link in the GWWC forum.

Here are some papers you should add to your bibliography, if you haven't already:

What is the Probability Your Vote Will Make a Difference? Voting as a Rational Choice

In the first paper, his probability estimate is 1 in 60 million on average for a voter in a US presidential election, 1 in 10 million in the best cases (New Mexico, Virginia, New Hampshire, and Colorado).

If you focused on the best case, that could mean an order of magnitude for you.

On this point, it is noteworthy that international health aid eliminated small pox. According to Toby Ord, it is estimated that this has prevented over 100 million deaths, which is more than the total number of people that died in all wars in the 20th century. If you assumed that all of the rest of international health aid achieved nothing at all, this single effort would make the average number of dollars per DALY achieved by international health aid better than what the British Government achieves.

Still don't get it. Let's say cards are being put in front of my face, and all I'm getting is their color. I can reliability distinguish the colors here "http://www.webspresso.com/color.htm". How do I associate a sequence of cards with a string? It doesn't seem like there is any canonical way of doing this. Maybe it won't matter that much in the end, but are there better and worse ways of starting?

Ok, but how?

Question about Solomonoff induction: does anyone have anything good to say about how to associate programs with basic events/propositions/possible worlds?

This doesn't sound like a bad idea. Could someone give reasons to think that donations to SIAI now would be better than this?

I think the page makes a case that it is worth doing something about AI risk, and that SIAI is doing something. The page gives no one any reason to think that SIAI is doing better than anything else you could do about x-risk (there could be reasons elsewhere).

In this respect, the page is similar to other non-profit pages: (i) argue that there is a problem, (ii) argue that you're doing something to solve the problem, but don't (iii) try to show that you're solving the problem better than others. Maybe that's reasonable, since that rubs some donors the wrong way and is hard to establish that you're the best; but it doesn't advance our discussion about the best way to reduce x-risk.

Thanks.

So, you're really interested in this question: what is the best decision algorithm? And then you're interested, in a subsidiary way, in what you ought to do. You think the "action" sense is silly, since you can't run one algorithm and make some other choice.

Your answer to my objection involving the parody argument is that you ought to do something else (not go with loss aversion) because there is some better decision algorithm (that you could, in some sense of "could", use?) that tells you to do something else.

What do you do with case... (read more)

tl;dr Philsophers have been writing about what probabilities reduce to for a while. As far as I know, the only major reductionist view is David Lewis's "best system" account of laws (of nature) and chances. You can look for "best system" in this article for an intro. Barry Loewer has developed this view in this paper.

I agree with all of this.

I agree that this fact [you can't have a one-boxing disposition and then two box] could appear as premise in an argument, together with an alternative proposed decision theory, for the conclusion that one-boxing is a bad idea. If that was the implicit argument, then I now understand the point.

To be clear: I have not been trying to argue that you ought to take two boxes in Newcomb's problem.

But I thought this fact [you can't have a one-boxing disposition and then two box] was supposed to be a part of an argument that did not use a decision theory as a prem... (read more)

What is false is that you ought to have disposition a and do B.

OK. So the argument is this one:

1. According to two-boxers, you ought to (i) have the disposition to one-box, and (ii) take two boxes.
2. It is impossible to do (i) and (ii).
3. Ought implies can.
4. So two-boxers are wrong.

But, on your use of "disposition", two-boxers reject 1. They do not believe that you should have a FAWS-disposition to one-box, since having a FAWS-disposition to one-box just means "actually taking one box, where this is not a result of randomness". Two-bo... (read more)

Everyone agrees about what the best disposition to have is. The disagreement is about what to do. I have uniformly meant "ought" in the action sense, not the dispositional sense. (FYI: this is always the sense in which philosophers (incl. Richard) mean "ought", unless otherwise specified.)

BTW: I still don't understand the relevance of the fact that it is impossible for people with one-boxing dispositions to two-box. If you don't like the arguments that I formalized for you, could you tell me what other premises you are using to reach your conclusion?

Whatever you actually do (modulo randomness) at time t, that's your one and only disposition vs X at time t.

Okay, I understand how you use the word "disposition" now. This is not the way I was using the word, but I don't think that is relevant to our disagreement. I hereby resolve to use the phrase "disposition to A" in the same way as you for the rest of our conversation.

I still don't understand how this point suggests that people with one-boxing dispositions ought not to two-box. I can only understand it in one way: as in the ar... (read more)