Gram, may I hijack your expertise? Game theory is something that I've always wanted to study formally. Can you recommend me some sources to learn about things like mechanism design?

In this case, we should really define "coercion". Could you please elaborate what you meant through that word?

One could argue, that if someone holds a gun to your head and demands your money, it's not coercion, just a game, where the expected payoff of not giving the money is smaller than the expected payoff of handing it over.

(Of course, I completely agree with your explanation about taxes. It's just the usage of "coercion" in the rest of your comment which seems a little odd)

I do not think that Gram_Stone is making the claim that fining or jailing those who do not pay their taxes is not coercion. Instead, I think that he is arguing that it is not the coercion per se that results in most people paying their taxes, but rather that (due to the coercion) failing to pay taxes does not have a favorable payoff, and that it is the unfavorable payoff that causes most people to pay their taxes. So, if there were some way to create favorable payoffs for desirable behavior without coercion, then this would work just as well as does using coercion.

Gram_Stone, please correct me if that is not accurate. Also, do you have any ideas as to how to make voluntary payment of taxes have a favorable payoff without using coercion?

It's nice to see that someone else has thought about this.

It's a popular rationalist pastime to try coming up with munchkin solutions to social dilemmas. A friend posed one such munchkin solution to me, and I thought he had an unrealistic idea of why regulations work, so I said to him:

Even though it's what you really want, I don't think the fact that you know everyone else will cooperate is the interesting thing per se about regulations, but that this is a consequence of the fact that you have decreased what was once the temptation payoff and thus constructed a different game. You have functionally reduced the expected payoff of the option "Don't pay taxes," by law. If you don't pay taxes, then you get fined or jailed. Now all players are playing a game where the Nash equilibrium is also Pareto optimal: Pay taxes or be fined or jailed. Clearly, one should pay taxes.

Now, ironically, this is good news if we want to cause better outcomes with less or no coercion, because it suggests that it is not coercion in itself that does the good work, but the fact that we have changed the payoffs to construct a different game; we can interpret coercion as just one instantiation of the general process by which 'inefficient games' become 'efficient games'. Coercion is perhaps a simple way to do the thing that all possible solutions to this problem seem to have in common, but there may be others that we can assume to syntactically change the payoffs in the way that coercion does, but which we may semantically interpret as something other than coercion.

A different time, a friend noticed that people building up trust seemed qualitatively similar to a Prisoner's Dilemma but couldn't see exactly how. I was like, "Have you heard of Stag Hunt? That's the whole reason Rousseau came up with it!" PD is just one kind of coordination game.

More generally, isn't it weird that the central objects of study in game theory, despite all of the formalization that has taken place since the beginning of the field, are remembered in the form of anecdotes?! You learn about the Stag Hunt and the Prisoner's Dilemma and Chicken and all other sorts of game, but there doesn't really seem to be any systematic notion of how different games are connected, or if any games are 'closer' to others in some sense (as our intuitions might suggest).

Meditations on Moloch was pretty but in the audience I coughed the words 'mechanism design'. It just seems like pointing out the mainstream academic work makes you boring when you're commenting on something poetic. You also might like Robinson and Goforth's Topology of the 2x2 Games. The math isn't that complex and it provides more insight than a barrage of anecdotes. Note that to my knowledge this is not taught in traditional game theory courses but probably should be one day. They refer to this general class of games as the 'social dilemmas', if I recall correctly.

It seems possible that you can increase someone's raw capability without making them reflective enough to get all of the important answers right. This would mean that rationality is not just a bonus.

Even if intelligence doesn't help at all for advancing FAI relative to AGI except via rationality, it still seems pretty unlikely that intelligence amplification would hurt, even if it doesn't lead to improvements in rationality. It's not like intelligence amplification would decrease rationality.

Also, I think the language of 'critical levels' is probably better than the language of 'acceleration' in this context. ... A large part of why I consider IA-first an alternative worth thinking about is not because I think it's likely to differentially affect technological development in the obvious way, but because we may be below some critical threshold of intelligence necessary to build FAI and thus IA-first would be preferable to AI-first because AI-first would almost certainly fail. This also is not a sure thing and I think also warrants investigation.

I disagree. The hypothesis that it is literally impossible to build FAI (but not AGI) without intelligence amplification first is merely the most extreme version of the hypothesis that intelligence amplification accelerates FAI relative to AGI, and I don't see why it would be more plausible than less extreme versions.

if there is no hardware overhang, and WBE begins as a monopoly in the way that nuclear weapons began, then the monopolist may have to choose between uploading a lone human, psychological effects be damned, and delaying the use of an IA technique that is actually available to them. It seems that you could allow the lone emulation to interact with biological humans, and perhaps even 'pause' itself so that it experiences a natural amount of subjective time during social interaction, but if you abuse this too much for the sake of maintaining the emulation's mental health, then you sacrifice the gains in subjective time.

If you can run an emulation at much faster than human speed, then you don't have a hardware overhang. The hardware to run 10 emulations at 1/10 the speed should cost about the same amount. If you really have a hardware overhang, then the emulations are running slower than humans, which also decreases how dangerous they could be. Alternatively, it's possible that no one bothers running an emulation until they can do so at approximately human speed, at which point the emulation would be able to socialize with biological humans.

Yes, embryo selection and other non-WBE intelligence amplification techniques would be useful in similar ways as applying evolutionary algorithms to emulations. I'd expect non-WBE intelligence amplification to typically have much lower risks, but also smaller effect size, and would be useful independently of the eventual arrival of WBE technology.

I'm fairly confident that intelligence enhancement would be good for our chances of future survival. I'm not convinced by the case for fast economic growth increasing risk much, and FAI is probably a more IQ-intensive problem than AGI is, so intelligence enhancement would likely accelerate FAI more than AGI even if it doesn't result in increased rationality as well (although it would be a bigger plus if it did).

As a side note, something I've always wondered about is how unusually long periods of subjective time and potential relative social isolation would affect the mental health of uploads of modern humans.

I doubt it would be a problem. Forager bands tended to be small, and if hardware to run uploads on is not the limiting factor to first creating them, then it will be feasible to run small groups of uploads together as soon as it is feasible to run a single upload.

It seems to me that embryo selection could be one way to increase the intelligence of the first generation of uploads without (initially) producing a great deal of defective minds of the sort that you mention in the article. WBE may take several decades to achieve, the embryos will take some time to grow into adult humans anyway, and it seems to me that a great deal of the defective minds in your scenario wouldn't be created because we're selecting for genes that go on to produce minds, as opposed to selecting directly for mind designs. (Please do correct me if you were only talking about selecting for genes as opposed to mind designs.)

It's worth noting that embryo selection seems to me a much less extreme version of intelligence amplification than what you have suggested, and even with embryo selection it seems that we run into some old questions about how IQ and rationality are related. As argued elsewhere, it may be that finding ways to amplify intelligence without understanding this relation between intelligence and rationality could actually increase risk, as opposed to mitigating it.

As a side note, something I've always wondered about is how unusually long periods of subjective time and potential relative social isolation would affect the mental health of uploads of modern humans.

Meal replacements really are awesome. I've been on Soylent 2.0 for several months. I don't have to cook, wash dishes, or refrigerate my food, it's shipped to my front door, all of my waste is recyclable (to my knowledge), I don't have opportunities to impulse buy any more, it's more nutritious than what I was eating before, I'm a vegetarian (vegan, even?) as a side effect, and it's hard to choke on it. Rosa Labs also just released a caffeinated version with coffee in it, which will replace my energy drink habit. (I should probably drop caffeine altogether, but I love it so much.)

Cons include people looking at me weird, possible nutrition-related black swans, jaw muscle atrophy (which may be a notable pro for any transwomen out there), and lack of fibrous material to chew on to clean out the fissures in my molars, which could increase risk of dental caries. I could pick up a gum habit for the last one; xylitol gum would probably be best as long as I'm trying to improve my oral health.

I'm pretty sure that you're actually asking some deep questions right now.

I'm not all that well-versed in epistemology or probability theory, but when you write:

A friend in academia suggested that this touches on a problem with Bayes priors that has not been settled.

I think this is a reference to the problem of priors.

I think 'a problem with Bayes priors that has not been settled' kind of understates the significance.

And:

The first issue is that there are infinite people who never existed and did not have a coin made. If I narrow it to historic figures who turned out not to exist and did not have a coin made it becomes possible but also becomes subjective as to whether someone actually thought they existed. For example, did people believe the Minotaur existed? Perhaps I should choose another filter instead of historic figure, like humans that existed. But picking and choosing the category is again so subjective. Someone may also argue that woman inequality back then was so great that the data should only look at men, as a woman’s chance of being portrayed on a coin was skewed in a way that isn’t applicable to men.

I believe this is referred to as the reference class problem. It seems that in a Bayesian framework, the reference class problem is something of a subproblem of the problem of priors. It seems that you're only trying to define a suitable reference class in the first place because you're trying to produce a reasonable prior.

It's my understanding that one approach to the problem of priors has been to come up with a 'universal' prior, a prior which is reasonable to adopt before making any observations of the world. One example is Solomonoff's algorithmic probability. It seems however than even this may not be a satisfactory solution, because this prior defies our intuition in some ways. For example, humans might find it intuitive to assign nonzero prior probability to uncomputable hypotheses (e.g. our physics involves hypercomputation), but algorithmic probability only assigns nonzero probability to computable hypotheses, an agent with this prior will never be able to have credence in uncomputable hypotheses. Another problem is that, with this prior, hypotheses are penalized for their complexity, but utility can grow far more quickly than complexity. Increasing the number of happy people in a program from 1000 people to 1,000,000 people seems to increase its utility a lot without increasing its complexity by much. Taking this up to larger and larger numbers that become difficult to intuitively comprehend, it may be that such a prior would result in agents whose decision-making is dominated by very improbable outcomes with very high utilities. We can also ask if this apparently absurd result is a point against Solomonoff induction or if it's a point against how humans think, but if we humans are thinking the right way, we still don't know what it is that's going right inside of our heads and how it compares to Solomonoff induction.

For any other readers, sorry if I'm mistaken on any of this, it is quite technical and I haven't really studied it. Do correct me if I've made a mistake.

Back to my point, I think that you accidentally hit upon a problem that doesn't seem to take too many prerequisites to initially run into, and that, after a bit of squinting, turns out to be way harder than it seems it should be at first glance, given the small number of prerequisites necessary to realize that the problem exists. Personally, I would consider the task of providing a complete answer to your questions an open research problem.

Stanford Encyclopedia of Philosophy's entry on Bayesian epistemology might be some good reading for this.

Given the OP and this, I thought that you might like this.

That's not too closely related to the OP in one sense, but I've been collecting what I might call 'stories of broken science,' and thought you might be doing the same thing for different reasons.