It's a leap of faith to suppose that even our universe, never mind levels I-III, is exhausted by its mathematical properties, as opposed to simply mathematically describable. And I don't really see what it buys you. I suppose it's equally a leap of faith to suppose that our universe has more properties than that, but I just prefer not to leap at all.

In that case, there would be severe principle-agent problems, given the disparity between power/intelligence of the trainer/AI systems and the users. If I was someone who couldn't directly control an AI using your scheme, I'd be very concerned about getting uneven trades or having my property expropriated outright by individual AIs or AI conspiracies, or just ignored and left behind in the race to capture the cosmic commons. I would be really tempted to try another AI design that does purport to have the AI serve my interests directly, even if that scheme is not as "safe".

Are these worse than the principal-agent problems that exist in any industrialized society? Most humans lack effective control over many important technologies, both in terms of economic productivity and especially military might. (They can't understand the design of a car they use, they can't understand the programs they use, they don't understand what is actually going on with their investments...) It seems like the situation is quite analogous.

Moreover, even if we could build AI in a different way, it doesn't seem to do anything to address the problem, since it is equally opaque to an end user who isn't involved in the AI development process. In any case, they are in some sense at the mercy of the AI developer. I guess this is probably the key point---I don't understand the qualitative difference between being at the mercy of the software developer on the one hand, and being at the mercy of the software developer + the engineers who help the software run day-to-day on the other. There is a slightly different set of issues for monitoring/law enforcement/compliance/etc., but it doesn't seem like a huge change.

(Probably the rest of this comment is irrelevant.)

To talk more concretely about mechanisms in a simple example, you might imagine a handful of companies who provide AI software. The people who use this software are essentially at the mercy of the software providers (since for all they know the software they are using will subvert their interests in arbitrary ways, whether or not there is a human involved in the process). In the most extreme case an AI provider could effectively steal all of their users' wealth. They would presumably then face legal consequences, which are not qualitatively changed by the development of AI if the AI control problem is solved. If anything we expect the legal system and government to better serve human interests.

We could talk about monitoring/enforcement/etc., but again I don't see these issues as interestingly different from the current set of issues, or as interestingly dependent on the nature of our AI control techniques. The most interesting change is probably the irrelevance of human labor, which I think is a very interesting issue economically/politically/legally/etc.

I agree with the general point that as technology improves a singleton becomes more likely. I'm agnostic on whether the control mechanisms I describe would be used by a singleton or by a bunch of actors, and as far as I can tell the character of the control problem is essentially the same in either case.

I do think that a singleton is likely eventually. From the perspective of human observers, a singleton will probably be established relatively shortly after wages fall below subsistence (at the latest). This prediction is mostly based on my expectation that political change will accelerate alongside technological change.

An interesting comment:

The European champion of Go is not the world champion, or even close. The BBC, for example, reported that “Google achieves AI ‘breakthrough’ by beating Go champion,” and hundreds of other news outlets picked up essentially the same headline. But Go is scarcely a sport in Europe; and the champion in question is ranked only #633 in the world. A robot that beat the 633rd-ranked tennis pro would be impressive, but it still wouldn’t be fair to say that it had “mastered” the game. DeepMind made major progress, but the Go journey is still not over; a fascinating thread at YCombinator suggests that the program — a work in progress — would currently be ranked #279.

Gwern has written an article for Wired, allegedly revealing the true identity of Satoshi Nakamoto:

http://www.wired.com/2015/12/bitcoins-creator-satoshi-nakamoto-is-probably-this-unknown-australian-genius/

Gwern has written an article for Wired, allegedly revealing the true identity of Satoshi Nakamoto:

http://www.wired.com/2015/12/bitcoins-creator-satoshi-nakamoto-is-probably-this-unknown-australian-genius/

This advice may go against other advice, but it's a tactic that has served me well: in making early-career decisions, such as your choice of major, always ask yourself which choice preserves future options.

For example, let's say you are considering a major, and you are equally interested in Architecture, Literature, and Engineering as careers.

Under my analysis, I would ask, which of these choices preserves the most options?

If you choose to pursue a degree in Literature, it is unlikely that you will be able to parley those skills into any kind of job in Architecture or in Engineering.

If you choose Architecture, you will find it very difficult (though not entirely impossible) to switch into Engineering for a graduate degree. However, you may find that you can try to pivot into some kind of Literary existence more easily.

If you choose Engineering, you'll find that Architectural schools will be eager to accept you for a graduate program, and the difficulty of switching from Engineering to a Literature program will probably be equal to the difficulty of switching from Architecture.

So, under this analysis, Engineering is the choice that preserves the most future options. At the point of choosing your college major, you're too young to be screening off future possibilities. Unless you're completely gung-ho about Literature, and feel a real certainty about what you want, it's best to keep more cards in your hand and let yourself make that exclusionary choice when you're older and wiser.

You may find that reading the classics in your spare time and writing a little bit of fiction now and then more than satisfies your Literary impulses, in which case, you'll be glad you didn't commit yourself to it as a career.

Conversely, if you commit to Engineering and find that you hate it, it's always easier to pivot to the other options.

As a general rule, things that are perceived as more difficult are easier to pivot away from, because the admissions gatekeepers for the perceived-as-less-difficult options will find you impressive due to where you're coming from. This heuristic is valid at all levels, for example, if you decide to go the Engineering route, choose the subdiscipline of Engineering that everybody else says is the hardest, scariest one. You can always punt to one of the easier ones if you don't find it to be a good fit, but it's much harder to go uphill from where you start.

Four years ago, I asked three members of my close family who were likely to give me something for Christmas to make a donation to GiveWell/AMF (GiveWell's top charity at that point) instead of getting something for me. This wasn't burdensome at all for me, because I didn't have many unmet material needs at this point.

Anyways, in my case, it turned out that my upper middle class American relatives, who were culturally "normal", rather than being culturally close to any EA/Silicon Valley/rationality circles, were quite offended by this suggestion. This may have had something to do with my presentation--I don't remember myself being particularly good at politics or speaking back then, and I tried to be nice, but perhaps I was too bold, or too culturally insensitive. Still, I was quite surprised at how poorly my request was received.

Of the three family members I talked to, two told me that I was being unrealistic, and that I needed to realize how things worked in the real world, or something like that. They got me something comparable to what they'd gotten me the previous year. The third one actually made a donation to the Carter Center, but only after bemoaning how I didn't appreciate how hard making money was in the real world--I think they had liked the Carter Center because a friend had worked there, or something.

A couple family members I hadn't talked to heard about my request, and I later heard that one had been talking with other members of my family about how she had become "very worried" that I was going to become "too altruistic". Another actually bought a chicken (or goat?) in my name through Heifer International. That was interesting, since I had thought that I been clear that I preferred GiveWell's top charities over other charities.

I guess that I completely stopped being vocal about EA after that point. Still, I've often wondered if the type of EAs who hold birthday and Christmas fundraisers are more, or less culturally normal-feeling to the average first-worlder than my family is.

(Also, since this comment is about how I'm terrible at understanding how to get the tone right on EA things, I apologize if the tone of this comment itself is somewhat off.)

Is there a deadline for when the survey will close?

I like this because it's something to point to when arguing with somebody with an obvious bias toward anthropomorphizing the agents.

You show them a model like this, then you say, "Oh, the agent can reduce its movement penalty if it first consumes this other orange glowing box. The orange glowing box in this case is 'humanity' but the agent doesn't care."

edit: Don't normally care about downvotes, but my model of LW does not predict 4 downvotes for this post, am I missing something?

I don't know if this solves very much. As you say, if we use the number 1, then we shouldn't wear seatbelts, get fire insurance, or eat healthy to avoid getting cancer, since all of those can be classified as Pascal's Muggings. But if we start going for less than one, then we're just defining away Pascal's Mugging by fiat, saying "this is the level at which I am willing to stop worrying about this".

Also, as some people elsewhere in the comments have pointed out, this makes probability non-additive in an awkward sort of way. Suppose that if you eat unhealthy, you increase your risk of one million different diseases by plus one-in-a-million chance of getting each. Suppose also that eating healthy is a mildly unpleasant sacrifice, but getting a disease is much worse. If we calculate this out disease-by-disease, each disease is a Pascal's Mugging and we should choose to eat unhealthy. But if we calculate this out in the broad category of "getting some disease or other", then our chances are quite high and we should eat healthy. But it's very strange that our ontology/categorization scheme should affect our decision-making. This becomes much more dangerous when we start talking about AIs.

Also, does this create weird nonlinear thresholds? For example, suppose that you live on average 80 years. If some event which causes you near-infinite disutility happens every 80.01 years, you should ignore it; if it happens every 79.99 years, then preventing it becomes the entire focus of your existence. But it seems nonsensical for your behavior to change so drastically based on whether an event is every 79.99 years or every 80.01 years.

Also, a world where people follow this plan is a world where I make a killing on the Inverse Lottery (rules: 10,000 people take tickets; each ticket holder gets paid $1, except a randomly chosen "winner" who must pay $20,000)