Cross-posted from MIRI's blog.

MIRI aims to do research now that increases humanity's odds of successfully managing important AI-related events that are at least a few decades away. Thus, we'd like to know: To what degree can we take actions now that will predictably have positive effects on AI-related events decades from now? And, which factors predict success and failure in planning for decades-distant events that share important features with future AI events?

Or, more generally: How effectively can humans plan for future decades? Which factors predict success and failure in planning for future decades?

To investigate these questions, we asked Jonah Sinick to examine historical attempts to plan for future decades and summarize his findings. We pre-committed to publishing our entire email exchange on the topic (with minor editing), just as Jonah had done previously with GiveWell on the subject of insecticide-treated nets. The post below is a summary of findings from our full email exchange (.docx) so far.

We decided to publish our initial findings after investigating only a few historical cases. This allows us to gain feedback on the value of the project, as well as suggestions for improvement, before continuing. It also means that we aren't yet able to draw any confident conclusions about our core questions.

The most significant results from this project so far are:

1. Jonah's initial impressions about The Limits to Growth (1972), a famous forecasting study on population and resource depletion, were that its long-term predictions were mostly wrong, and also that its authors (at the time of writing it) didn't have credentials that would predict forecasting success. Upon reading the book, its critics, and its defenders, Jonah concluded that many critics and defenders had  seriously misrepresented the book, and that the book itself exhibits high epistemic standards and does not make significant predictions that turned out to be wrong.
2. Svante Arrhenius (1859-1927) did a surprisingly good job of climate modeling given the limited information available to him, but he was nevertheless wrong about two important policy-relevant factors. First, he failed to predict how quickly carbon emissions would increase. Second, he predicted that global warming would have positive rather than negative humanitarian impacts. If more people had taken Arrhenius' predictions seriously and burned fossil fuels faster for humanitarian reasons, then today's scientific consensus on the effects of climate change suggests that the humanitarian effects would have been negative.
3. In retrospect, Norbert Wiener's concerns about the medium-term dangers of increased automation appear naive, and it seems likely that even at the time, better epistemic practices would have yielded substantially better predictions.
4. Upon initial investigation, several historical cases seemed unlikely to shed substantial light on our  core questions: Norman Rasmussen's analysis of the safety of nuclear power plants, Leo Szilard's choice to keep secret a patent related to nuclear chain reactions, Cold War planning efforts to win decades later, and several cases of "ethically concerned scientists."
5. Upon initial investigation, two historical cases seemed like they might shed light on our  core questions, but only after many hours of additional research on each of them: China's one-child policy, and the Ford Foundation's impact on India's 1991 financial crisis.
6. We listed many other historical cases that may be worth investigating.

The project has also produced a chapter-by-chapter list of some key lessons from Nate Silver's The Signal and the Noise, available here.

Further details are given below. For sources and more, please see our full email exchange (.docx).

Less Wrong is based on reddit code, which means we can create subreddits with relative ease.

Right now we have two subreddits, Main and Discussion. These are distinguished not by subject matter, but by whether a post is the type of thing that might be promoted to the front page or not (e.g. a meetup announcement, or a particularly well-composed and useful post).

As a result, almost everything is published to Discussion, and thus it is difficult for busy people to follow only the subjects they care about. More people will be able to engage if we split things into topic-specific subreddits, and make it easy to follow only what they care about.

To make it easier for people to follow only what they care about, we're building the code for a Dashboard thingie.

But we also need to figure out which subreddits to create, and we'd like community feedback about that.

We'll probably start small, with just 1-5 new subreddits.

Below are some initial ideas, to get the conversation started.

Idea 1

• Main: still the place for things that might be promoted.
• Applied Rationality: for articles about what Jonathan Baron would call descriptive and prescriptive rationality, for both epistemic and instrumental rationality (stuff about biases, self-improvement stuff, etc.).
• Normative Rationality: for articles about what Baron would call normative rationality, for both epistemic and instrumental rationality (examining the foundations of probability theory, decision theory, anthropics, and lots of stuff that is called "philosophy"). 
• The Future: for articles about forecasting, x-risk, and future technologies.
• Misc: Discussion, renamed, for everything that doesn't belong in the other subreddits.

Idea 2

• Main
• Epistemic Rationality: for articles about how to figure out the world, spanning the descriptive, prescriptive, and normative.
• Instrumental Rationality: for articles about how to take action to achieve your goals, spanning the descriptive, prescriptive, and normative. (One difficulty with the epistemic/instrumental split is that many (most?) applied rationality techniques seem to be relevant to both epistemic and instrumental rationality.)
• The Future
• Misc.

Bolonkin & Friedlander (2013) argues that it might be possible for "a dying dictator" to blow up the Sun, and thus destroy all life on Earth:

The Sun contains ~74% hydrogen by weight. The isotope hydrogen-1 (99.985% of hydrogen in nature) is a usable fuel for fusion thermonuclear reactions. This reaction runs slowly within the Sun because its temperature is low (relative to the needs of nuclear reactions). If we create higher temperature and density in a limited region of the solar interior, we may be able to produce self-supporting detonation thermonuclear reactions that spread to the full solar volume. This is analogous to the triggering mechanisms in a thermonuclear bomb. Conditions within the bomb can be optimized in a small area to initiate ignition, then spread to a larger area, allowing producing a hydrogen bomb of any power. In the case of the Sun certain targeting practices may greatly increase the chances of an artificial explosion of the Sun. This explosion would annihilate the Earth and the Solar System, as we know them today. The reader naturally asks: Why even contemplate such a horrible scenario? It is necessary because as thermonuclear and space technology spreads to even the least powerful nations in the centuries ahead, a dying dictator having thermonuclear missile weapons can [produce] (with some considerable mobilization of his military/industrial complex)—an artificial explosion of the Sun and take into his grave the whole of humanity. It might take tens of thousands of people to make and launch the hardware, but only a very few need know the final targeting data of what might be otherwise a weapon purely thought of (within the dictator’s defense industry) as being built for peaceful, deterrent use. Those concerned about Man’s future must know about this possibility and create some protective system—or ascertain on theoretical grounds that it is entirely [impossible]. Humanity has fears, justified to greater or lesser degrees, about asteroids, warming of Earthly climate, extinctions, etc. which have very small probability. But all these would leave survivors—nobody thinks that the terrible annihilation of the Solar System would leave a single person alive. That explosion appears possible at the present time. In this paper is derived the “AB-Criterion” which shows conditions wherein the artificial explosion of Sun is possible. The author urges detailed investigation and proving or disproving of this rather horrifying possibility, so that it may be dismissed from mind—or defended against.

Warning: the paper is published in an obscure journal by publisher #206 on Beall’s List of Predatory Publishers 2013, and I was unable to find confirmation of the authors' claimed credentials from any reputable sources with 5 minutes of Googling. It also has two spelling errors in the abstract. (It has no citations on Google scholar, but I wouldn't expect it to have any since it was only released in July 2013.)

I haven't read the paper, and I'd love to see someone fluent in astrophysics comment on its contents. 

My guess is that this is not a risk at all or, as with proposed high-energy physics disasters, the risk is extremely low-probability but physically conceivable (though perhaps not by methods imagined by Bolonkin & Friedlander). 

I've just posted an analysis to MIRI's blog called Transparency in Safety-Critical Systems. Its aim is to explain a common view about transparency and system reliability, and then open a dialogue about which parts of that view are wrong, or don't apply well to AGI.

The "common view" (not universal by any means) explained in the post is, roughly:

Black box testing can provide some confidence that a system will behave as intended, but if a system is built such that it is transparent to human inspection, then additional methods of reliability verification are available. Unfortunately, many of AI’s most useful methods are among its least transparent. Logic-based systems are typically more transparent than statistical methods, but statistical methods are more widely used. There are exceptions to this general rule, and some people are working to make statistical methods more transparent.

Three caveats / open problems listed at the end of the post are:

1. How does the transparency of a method change with scale? A 200-rules logical AI might be more transparent than a 200-node Bayes net, but what if we’re comparing 100,000 rules vs. 100,000 nodes? At least we can query the Bayes net to ask “what it believes about X,” whereas we can’t necessarily do so with the logic-based system.
2. Do the categories above really “carve reality at its joints” with respect to transparency? Does a system’s status as a logic-based system or a Bayes net reliably predict its transparency, given that in principle we can use either one to express a probabilistic model of the world?
3. How much of a system’s transparency is “intrinsic” to the system, and how much of it depends on the quality of the user interface used to inspect it? How much of a “transparency boost” can different kinds of systems get from excellently designed user interfaces?

The MIRI blog has only recently begun to regularly host substantive, non-news content, so it doesn't get much commenting action yet. Thus, I figured I'd post here and try to start a dialogue. Comment away!

When I talk about the poor distribution of funds in charity, people in the effective altruism movement sometimes say, "Didn't Holden Karnofsky show that charity is an efficient market in his post Broad Market Efficiency?"

My reply is "No. Holden never said, and doesn't believe, that charity is an efficient market."

What is an efficient market?

An efficient market is one in which "one cannot consistently achieve returns in excess of average market returns... given the information available at the time the investment is made." (Details here.)

Of course, market efficiency is a spectrum, not a yes/no question. As Holden writes, "The most efficient markets can be consistently beaten only by the most talented/dedicated players, while the least efficient [markets] can be beaten with fairly little in the way of talent and dedication."

Moreover, market efficiency is multi-dimensional. Any particular market may be efficient in some ways, and in some domains, while highly inefficient in other ways and other domains.

Is Less Wrong, despite its flaws, the highest-quality relatively-general-interest forum on the web? It seems to me that, to find reliably higher-quality discussion, I must turn to more narrowly focused sites, e.g. MathOverflow and the GiveWell blog.

Many people smarter than myself have reported the same impression. But if you know of any comparably high-quality relatively-general-interest forums, please link me to them!

In the meantime: suppose it's true that Less Wrong is the highest-quality relatively-general-interest forum on the web. In that case, we're sitting on a big opportunity to grow Less Wrong into the "standard" general-interest discussion hub for people with high intelligence and high metacognition (shorthand: "intellectual elites").

Earlier, Jonah Sinick lamented the scarcity of elites on the web. How can we get more intellectual elites to engage on the web, and in particular at Less Wrong?

Some projects to improve the situation are extremely costly:

1. Pay some intellectual elites with unusually good writing skills (like Eliezer) to generate a constant stream of new, interesting content.
2. Comb through Less Wrong to replace community-specific jargon with more universally comprehensible terms, and change community norms about jargon. (E.g. GiveWell's jargon tends to be more transparent, such as their phrase "room for more funding.")

Code changes, however, could be significantly less costly. New features or site structure elements could increase engagement by intellectual elites. (To avoid priming and contamination, I'll hold back from naming specific examples here.)

To help us figure out which code changes are most likely to increase engagement on Less Wrong by intellectual elites, specific MIRI volunteers will be interviewing intellectual elites who (1) are familiar enough with Less Wrong to be able to simulate which code changes might cause them to engage more, but who (2) mostly just lurk, currently.

In the meantime, I figured I'd throw these ideas to the community for feedback and suggestions.

Today MIRI released a new technical report by visiting researcher Katja Grace called "Algorithmic Progress in Six Domains." The report summarizes data on algorithmic progress – that is, better performance per fixed amount of computing hardware – in six domains:

• SAT solvers,
• Chess and Go programs,
• Physics simulations,
• Factoring,
• Mixed integer programming, and
• Some forms of machine learning.

MIRI's purpose for collecting these data was to shed light on the question of intelligence explosion microeconomics, though we suspect the report will be of broad interest within the software industry and computer science academia.

One finding from the report was previously discussed by Robin Hanson here. (Robin saw an early draft on the intelligence explosion microeconomics mailing list.)

This is the preferred page for discussing the report in general.

Summary:

In recent boolean satisfiability (SAT) competitions, SAT solver performance has increased 5–15% per year, depending on the type of problem. However, these gains have been driven by widely varying improvements on particular problems. Retrospective surveys of SAT performance (on problems chosen after the fact) display significantly faster progress.

Chess programs have improved by around 50 Elo points per year over the last four decades. Estimates for the significance of hardware improvements are very noisy, but are consistent with hardware improvements being responsible for approximately half of progress. Progress has been smooth on the scale of years since the 1960s, except for the past five. Go programs have improved about one stone per year for the last three decades. Hardware doublings produce diminishing Elo gains, on a scale consistent with accounting for around half of progress.

Improvements in a variety of physics simulations (selected after the fact to exhibit performance increases due to software) appear to be roughly half due to hardware progress.

The largest number factored to date has grown by about 5.5 digits per year for the last two decades; computing power increased 10,000-fold over this period, and it is unclear how much of the increase is due to hardware progress.

Some mixed integer programming (MIP) algorithms, run on modern MIP instances with modern hardware, have roughly doubled in speed each year. MIP is an important optimization problem, but one which has been called to attention after the fact due to performance improvements. Other optimization problems have had more inconsistent (and harder to determine) improvements.

Various forms of machine learning have had steeply diminishing progress in percentage accuracy over recent decades. Some vision tasks have recently seen faster progress.