Seriously! I just overheard someone say "wow, maybe all that rationality stuff actually does help them do better."

2009: "Extreme Rationality: It's Not That Great"

2010: "Self-Improvement or Shiny Distraction: Why Less Wrong is anti-Instrumental Rationality"

2013: "How about testing our ideas?"

2014: "Truth: It's Not That Great"

2015: "Meta-Countersignaling Equilibria Drift: Can We Accelerate It?"

2016: "In Defense Of Putting Babies In Wood Chippers"

It seemed pretty obvious to me that MIRI thinks defenses cannot be made, whether or not such a list exists, and wants easier ways to convince people that defenses cannot be made. Thus the part that said: "We would especially like suggestions which are plausible given technology that normal scientists would expect in the next 15 years. So limited involvement of advanced nanotechnology and quantum computers would be appreciated. "

It seemed pretty obvious to me that the point of making such a list was to plan defenses.

So MIRI is interested in making a better list of possible concrete routes to AI taking over the world.

I wouldn't characterize this as something that MIRI wants.

To clarify, One Medical partnered with us on this event... but are not materially involved with expanding MIRI themselves. They're simply an innovative business nearby us in Berkeley who wanted to support our work. I know it's somewhat unprecedented to see MIRI with strong corporate support, but trust me, it's a good thing. One Medical's people did a ton of legwork and made it super easy to host over 100 guests at that event with almost no planning needed on our part. They took care of everything so we could just focus on our work. A perfect partnership in our opinion.

Also, we still have $149 credits for the free 1-year memberships to One Medical's service. If you live in Berkeley, SF, NY, Boston, Chicago, LA, or DC and are looking for a good primary care doctor, check out their website and if you think it's a good fit for you, take them up on their promotional offer with this link: http://bit.ly/1fnRHrH (expires 4/9/14).

The material covered in Causality is more like a subset of that in PGM. PGM is like an encyclopedia, and Causality is a comprehensive introduction to one application of PGMs.

I noticed that the course list doesn't cover several topics that are popular on LW. Some suggestions:

Game theory - Fudenberg and Tirole

K-complexity - Li and Vitanyi

Causality - Pearl

And maybe something on cryptography, but I don't know enough about it to recommend a good book.

The links to Eliezer's Open Problems in FAI papers are broken.

A question at this point I might ask is how good does the final estimate have to be?

First, there are multiple applications of accurate estimates.

The unreasonably low estimates would suggest things like "I'm net reducing factory-farming suffering if I eat meat and donate a few bucks, so I should eat meat if it makes me happier or healthier sufficiently to earn and donate an extra indulgence of $5 ."

There are some people going around making the claim, based on the extreme low-ball cost estimates, that these veg ads would save human lives more cheaply than AMF by reducing food prices. With saner estimates, not so, I think.

Second, there's the question of flow-through effects, which presumably dominate in a total utilitarian calculation anyway, if that's what you're into. The animal experiences probably don't have much effect there, but people being vegetarian might have some, as could effects on human health, pollution, food prices, social movements, etc.

To address the total utilitarian question would require a different sort of evidence, at least in the realistic ranges.