'Let an ultraintelligent person be defined as a person who can far surpass all the intellectual activities of any other person however clever. Since the improvement of people is one of these intellectual activities, an ultraintelligent person could produce even better people; there would then unquestionably be an 'intelligence explosion,' and the intelligence of ordinary people would be left far behind. Thus the first ultraintelligent person is the last invention that people need ever make, provided that the person is docile enough to tell us how to keep them under control.'

Does this work?

This is awesomely paranoid. Thank you for pointing this out.

I'm a little worried a solution here will call for whoever controls the webapp to also be an expert at creating placebos for every product type. (If we trust contract manufacturers to be honest, then the issue of adding poisons to a placebo can be handled by having them ship directly to the third party for mailing... but I that's already the default case).

Perhaps poisons can be discovered by looking at other products which performed the same protocol? "This experiment has to be re-done because the control group mysteriously got sick" doesn't seem like a good solution though...

I'll wrestle with this. Maybe something with MaxL's answer to #8 might be possible?

There could be more or fewer of various parts; I could not link to so many things if nobody actually wants to pursue things to greater depth; the questions could be different in level or kind; the language could be suited to a different audience; we could have an online meetup to talk about the most interesting things; I could try to interview a relevant expert and post it; I could post a multiple choice test to see if you remember the material; the followup research questions could be better suited for an afternoon rather than a PhD...

How would you like this reading group to be different in future weeks?

Bostrom talks about a seed AI being able to improve its 'architecture', presumably as opposed to lower level details like beliefs. Why would changing architecture be particularly important?

This seems like an information hazard, since it has the form: This estimate for process X which may destroy the value of the future seems too low, also, not many people are currently studying X, which is surprising.

If X is some genetically engineered variety of smallpox, it seems clear that mentioning those facts is hazardous.

If the World didn't know about brain emulation, calling it an under-scrutinized area would be dangerous, relative to, say, just mention to a few safety savvy, x-risk savvy neuroscientist friends who would go on to design safety protocols for it, as well as, if possible, slow down progress in the field.

Same should be done in this case.

I am intrigued by the estimate for the difficulty of recapitulating evolution. Bostrom estimates 1E30 to 1E40 FLOPSyears. A conservative estimate for the value of a successful program to recapitulate evolution might be around $500B. This is enough to buy something like 10k very large supercomputers for a year, which gets you something like 1E20 FLOPSyears. So the gap is between 10 and 20 orders of magnitude. In 35 years, this gap would fall to 5 to 15 orders of magnitude (at the current rate of progress in hardware, which seems likely to slow).

One reason this possibility is important is that it seems to offer one of the strongest possible environments for a disruptive technological change.

This seems sensible as a best guess, but it is interesting to think about scenarios where it turns out to be surprisingly easy to simulate evolution. For example, if there were a 10% chance of this project being economically feasible within 20 years, that would be an extremely interesting fact, and one that might affect my views of the plausibility of AI soon. (Not necessarily because such an evolutionary simulation per se is likely to occur, but mostly because it says something about the overall difficulty of building up an intelligence by a brute force search.)

But it is easy to see how this estimate might be many orders of magnitude too high (and also to see how it could be somewhat quite a bit too low, but it is interesting to look at the low tail in particular):

• It may be that you can memoize much of the effort of fitness evaluations, evaluating the fitness of many similar organisms in parallel. This appears to be a trick that is unavailable to evolution, but the key idea on which modern approaches to training neural nets rely. Gradient descent + backpropagation gives you a large speedup for training neural nets, as much as linear in the number of parameters which you are training. In the case of evolution, this could already produce a 10 order of magnitude speedup.
• It may be possible to evaluate the fitness of an organism radically faster than it is revealed by nature. For example, it would not be too surprising if you could short-circuit most of the work of development. Moreover, nature doesn't get very high-fidelity estimates of fitness, and it wouldn't be at all surprising to me if it were possible to get a comparably good estimate of a human's fitness over the course of an hour in a carefully designed environment (this is a speedup of about 5 orders of magnitude over the default).
• It seems plausible that mutation in nature does not search the space as effectively as human engineers could even with a relatively weak understanding of intelligence or evolution. It would not be surprising to me if you could conduct the search a few orders of magnitude faster merely by using optimal mutation rates, using estimates for fitness including historical performance, choosing slightly better distribution of mutations, or whatever.

Overall, I would not be too surprised (certainly I would give it > 1%) if a clean theoretical understanding of evolution made this a tractable project today given sufficient motification, which is quite a frightening prospect. The amount of effort that has gone into developing such an understanding, with an eye towards this kind of engineering project, also appears to be surprisingly small.

Why do you think the scale of the bias is unlikely to be more than a few decades?

Because the differences between estimates made by people who should be highly selected for optimism (e.g. AGI researchers) and people who should be much less so (other AI researchers, and more importantly but more noisily, other people) are only a few decades.

Some thoughts on this perspective:

1. Most people are not so exclusively interested in existential risk reduction; their decisions depend on how the development of AI compares to more pressing concerns. I think you can make a good case that normal humanitarians are significantly underestimating the likely impact of AI; if that's true, then by making that case one might be able to marshall a lot of additional effort.

2. Echoing Katja: general improvements in individual and collective competence are also going to have a material effect on how the development of AI is handled. If AI is far off (e.g. if we were having this discussion in 1600) then it seems that those effects will tend to dominate the achievable direct impacts. Even if AI is developed relatively soon, it's still plausible to me that institutional quality will be a big determinant of outcomes relative to safety work (though it's less plausible on the margin, given just how little safety work there is).

I can imagine a future where all of the low-hanging fruit is taken in many domains, so that the best available interventions for altrusits concerned with long-term trajectories is focusing on improbable scenarios that are being neglected by the rest of the world because they don't care as much. For better or worse, I don't think we are there yet.

I agree with the general sentiment. Though if human-level AI is very far away, I think there might be better things to do now than work on very direct safety measures. For instance, improve society's general mechanisms for dealing with existential risks, or get more information about what's going to happen and how to best prepare. I'm not sure if you meant to include these kinds of things.