In the past year I've been involved in two major projects at SIAI. Steve Rayhawk and I were asked to review existing AGI literature and produce estimates of development timelines for AGI.

You seem to suggest that this work is incomplete, but I'm curious: is this available anywhere or is it still a work in-progress? I would be very interested in reading this, even if its incomplete. I would even be interested in just seeing a bibliography.

I wrote earlier:

Where I've seen people use PDUs in AI or philosophy, they weren't confused, but rather chose to make the assumption of perception-determined utility functions (or even more restrictive assumptions) in order to prove some theorems.

Well, here's a recent SIAI paper that uses perception-determined utility functions, but apparently not in order to prove theorems (since the paper contains no theorems). The author was advised by Peter de Blanc, who two years ago wrote the OP arguing against PDUs. Which makes me confused: does the author (Daniel Dewey) really think that PDUs are a good idea, and does Peter now agree?

"I am a god" is to simplistic. I can model it better as a probability, that varies with N, that you are able to move the universe to UN(N). This tracks how good a god you are, and seems to make the paradox disappear.

LCPW isn't even necessary: do you really think that it wouldn't make a difference that you'd care about?

resources, whether physical or computational. Presumably the AI is programmed to utilize resources in a parsimonious manner, with terms governing various applications of the resources, including powering the AI, and deciding on what to do. If the AI is programmed to limit what it does at some large but arbitrary point, because we don't want it taking over the universe or whatever, then this point might end up actually being before we want it to stop doing whatever it's doing.

A small risk of losing the utility it was previously counting on.

Of course you can do intuition pumps either way- I don't feel like I'd want the AI to sacrifice everything in the universe we know for a 0.01% chance of making it in a bigger universe- but some level of risk has to be worth a vast increase in potential fun.

Upvoted because the objection makes me uncomfortable, and because none of the replies satisfy my mathematical/aesthetic intuition.

However, requiring utilities to be bounded also strikes me as mathematically ugly and practically dangerous– what if the universe turns out to be much larger than previously thought, and the AI says "I'm at 99.999% of achievable utility already, it's not worth it to expand farther or live longer"?

Thus I view this as a currently unsolved problem in decision theory, and a better intuition-pump version than Pascal's Mugging. Thanks for posting.

1. A physically plausible scenario would involve growing up under a monochromatic light source.

2. Growing up without sensory input actually affects the brain; see Wikipedia's article on monocular deprivation. I'm actually an example of this - I was born without the Mystic Eyes Of Depth Perception so I'll never know what stereoscopic vision "feels like".

3. I propose that "qualia" is a word that, like "microevolution", is mainly used by people who are very confused (and dissolving the question is the appropriate approach).

"Rational people can't agree to disagree" is an oversimplification. Rational people can perfectly well reach a conclusion of the form: "Our disagreement on this matter is a consequence of our disagreement on other issues that would be very difficult to resolve, and for which there are many apparently intelligent, honest and well informed people on both sides. Therefore, it seems likely that reaching agreement on this issue would take an awful lot of work and wouldn't be much more likely to leave us both right than to leave us both wrong. We choose, instead, to leave the matter unresolved until either it matters more or we see better prospects of resolving it."

Imperfectly rational people who are aware of their imperfect rationality (note: this is in fact the nearest any of us actually come to being rational people) might also reasonably reach a conclusion of this form: "Perhaps clear enough thinking on both sides would suffice to let us resolve this. However, it's apparent that at least one of us is currently sufficiently irrational about it that trying to reach agreement poses a real danger of spoiling the good relations we currently enjoy, and while clearly that irrationality is a bad thing it doesn't seem likely that trying to resolve our current disagreement now is the best way to address it, so let's leave it for now."

I suspect (with no actual evidence) that when two reasonably-rational people say they're agreeing to disagree, what they mean is often approximately one of the above or a combination thereof, and that they're often wise to "agree to disagree". The fact that there are theorems saying that two perfect rationalists who care about nothing more than getting the right answer to the question they're currently disputing won't "agree to disagree" seems to me to have little bearing on this.

Eliezer, if you're reading this: You may remember that a while back on OB you and Robin Hanson discussed the prospects of rapidly improving artificial intelligence in the nearish future. By no means did you resolve your differences in that discussion. Would it be fair to characterize the way it ended as "agreeing to disagree"? From the outside, it sure looks like that's what it amounted to, whatever you may or may not have said to one another about it. Perhaps you and/or Robin might say "Yeah, but the other guy isn't really rational about this". Could be, but if the level of joint rationality required for "can't agree to disagree" is higher than that of {Eliezer,Robin} then it's not clear how widely applicable the principle "rational people can't agree to disagree" really is. (Note for the avoidance of doubt: The foregoing is not intended to imply that Eliezer and Robin are equally rational; I do not intend to make any further comment on my opinions, if any, on that matter.)

Obviously I didn't mean that being broke (or anything) is infinite disutility. Am I mistaken that the utility of money is otherwise modeled as logarithmic generally?