Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

New report: Intelligence Explosion Microeconomics

44 Post author: Eliezer_Yudkowsky 29 April 2013 11:14PM

SummaryIntelligence Explosion Microeconomics (pdf) is 40,000 words taking some initial steps toward tackling the key quantitative issue in the intelligence explosion, "reinvestable returns on cognitive investments": what kind of returns can you get from an investment in cognition, can you reinvest it to make yourself even smarter, and does this process die out or blow up? This can be thought of as the compact and hopefully more coherent successor to the AI Foom Debate of a few years back.

(Sample idea you haven't heard before:  The increase in hominid brain size over evolutionary time should be interpreted as evidence about increasing marginal fitness returns on brain size, presumably due to improved brain wiring algorithms; not as direct evidence about an intelligence scaling factor from brain size.)

I hope that the open problems posed therein inspire further work by economists or economically literate modelers, interested specifically in the intelligence explosion qua cognitive intelligence rather than non-cognitive 'technological acceleration'.  MIRI has an intended-to-be-small-and-technical mailing list for such discussion.  In case it's not clear from context, I (Yudkowsky) am the author of the paper.

Abstract:

I. J. Good's thesis of the 'intelligence explosion' is that a sufficiently advanced machine intelligence could build a smarter version of itself, which could in turn build an even smarter version of itself, and that this process could continue enough to vastly exceed human intelligence.  As Sandberg (2010) correctly notes, there are several attempts to lay down return-on-investment formulas intended to represent sharp speedups in economic or technological growth, but very little attempt has been made to deal formally with I. J. Good's intelligence explosion thesis as such.

I identify the key issue as returns on cognitive reinvestment - the ability to invest more computing power, faster computers, or improved cognitive algorithms to yield cognitive labor which produces larger brains, faster brains, or better mind designs.  There are many phenomena in the world which have been argued as evidentially relevant to this question, from the observed course of hominid evolution, to Moore's Law, to the competence over time of machine chess-playing systems, and many more.  I go into some depth on the sort of debates which then arise on how to interpret such evidence.  I propose that the next step forward in analyzing positions on the intelligence explosion would be to formalize return-on-investment curves, so that each stance can say formally which possible microfoundations they hold to be falsified by historical observations already made.  More generally, I pose multiple open questions of 'returns on cognitive reinvestment' or 'intelligence explosion microeconomics'.  Although such questions have received little attention thus far, they seem highly relevant to policy choices affecting the outcomes for Earth-originating intelligent life.

The dedicated mailing list will be small and restricted to technical discussants.

This topic was originally intended to be a sequence in Open Problems in Friendly AI, but further work produced something compacted beyond where it could be easily broken up into subposts.

Outline of contents:

1:  Introduces the basic questions and the key quantitative issue of sustained reinvestable returns on cognitive investments.

2:  Discusses the basic language for talking about the intelligence explosion, and argues that we should pursue this project by looking for underlying microfoundations, not by pursuing analogies to allegedly similar historical events.

3:  Goes into detail on what I see as the main arguments for a fast intelligence explosion, constituting the bulk of the paper with the following subsections:

  • 3.1: What the fossil record actually tells us about returns on brain size, given that most of the difference between Homo sapiens and Australopithecus was probably improved software.
  • 3.2: How to divide credit for the human-chimpanzee performance gap between "humans are individually smarter than chimpanzees" and "the hominid transition involved a one-time qualitative gain from being able to accumulate knowledge".
  • 3.3: How returns on speed (serial causal depth) contrast with returns from parallelism; how faster thought seems to contrast with more thought.  Whether sensing and manipulating technologies are likely to present a bottleneck for faster thinkers, or how large of a bottleneck.
  • 3.4 How human populations seem to scale in problem-solving power; some reasons to believe that we scale inefficiently enough for it to be puzzling.  Garry Kasparov's chess match vs. The World, which Kasparov won.
  • 3.5 Some inefficiencies that might cumulate in an estimate of humanity's net computational efficiency on a cognitive problem.
  • 3.6 What the anthropological record actually tells us about cognitive returns on cumulative selection pressure, given that selection pressures were probably increasing over the course of hominid history.  How the observed history would be expected to look different, if there were in fact diminishing returns on cognition.
  • 3.7 How to relate the curves for evolutionary difficulty, human-engineering difficulty, and AI-engineering difficulty, considering that they are almost certainly different.
  • 3.8 Correcting for anthropic bias in trying to estimate the intrinsic 'difficulty 'of hominid-level intelligence just from observing that intelligence evolved here on Earth.
  • 3.9 The question of whether to expect a 'local' (one-project) FOOM or 'global' (whole economy) FOOM and how returns on cognitive reinvestment interact with that.
  • 3.10 The great open uncertainty about the minimal conditions for starting a FOOM; why I. J. Good's postulate of starting from 'ultraintelligence' is probably much too strong (sufficient, but very far above what is necessary).
  • 3.11 The enhanced probability of unknown unknowns in the scenario, since a smarter-than-human intelligence will selectively seek out and exploit flaws or gaps in our current knowledge.

4:  A tentative methodology for formalizing theories of the intelligence explosion - a project of formalizing possible microfoundations and explicitly stating their alleged relation to historical experience, such that some possibilities can allegedly be falsified.

5:  Which open sub-questions seem both high-value and possibly answerable.

6:  Formally poses the Open Problem and mentions what it would take for MIRI itself to directly fund further work in this field.

Comments (244)

Comment author: mwengler 29 April 2013 03:26:43PM 19 points [-]

Just a thought on chess playing. Rather than looking at an extreme like Kasparov vs the world, it would be interesting to me to have teams of two, three, and four players of well-known individual ranking. These teams could then play many games against individuals and against each other. The effective ranking of the teams could be determined from their results. In this way, some sense of "how much smarter" a team is than the individual members could be determined. Ideally, the team would not be ranked until it had had significant experience playing as a team. We are interested in what a team could accomplish, and no strong reason to think it would take less time to optimize a team than to optimize an individual.

Along the same lines, teams could be developed to take IQ and other GI correlated tests to see how much smarter a few people together are than a single human. Would the results have implications for optimal AI design?

Comment author: ThrustVectoring 29 April 2013 05:23:31PM 2 points [-]

As far as empirically finding the optimum group size, it'd be cheaper to find the number of researchers in a scientific sub-discipline and measure the productive work they do in that field. They are teams that review work for general distribution, read on others' progress, and contribute to the discussion. Larger sub-fields that would be more efficient divided up would have large incentives to do so, as defectors to the sub-sub-field would have higher productivity (and less irrelevant work to read up on).

Comment author: Eliezer_Yudkowsky 29 April 2013 04:25:10PM 4 points [-]

I think that teams of up to five people can scale "pretty well by human standards" - not too far from linearly. It's going up to a hundred, a thousand, a million, a billion that we start to run into incredibly sublinear returns.

Comment author: ThrustVectoring 29 April 2013 05:23:29PM 5 points [-]

As group size increases you have to spend more and more of your effort getting your ideas heard and keeping up with the worthwhile ideas being proposed by other people, as opposed to coming up with your own good ideas.

Depending on the relevant infrastructure and collaboration mechanisms, it's fairly easy to have a negative contribution from each additional person in the project. If someone is trying to say something, then someone else has to listen - even if all the listener does is keep it from lowering the signal-to-noise ratio by removing the contribution.

Comment author: DanArmak 29 April 2013 07:50:22PM 11 points [-]

You correctly describe the problems of coordinating the selection of the best result produced. But there's another big problem: coordinating the division of work.

When you add another player to a huge team of 5000 people, he won't start exploring a completely new series of moves no-one else had considered before. Instead, he will likely spend most of his time considering moves already considered by some of the existing players. That's another reason why his marginal contribution will be so low.

Unlike humans, computers are good at managing divide-and-conquer problems. In chess, a lot of the search for the next move is local in the move tree. That's what makes it a particularly good example of human groups not scaling where computers would.

Comment author: mwengler 29 April 2013 10:12:40PM 2 points [-]

Are you or anyone else aware of any work along these lines, showing the intelligence of groups of people?

Any sense of what the intelligence of the planet as a whole, or the largest effective intelligence of any group on the planet might be?

If groups of up to 5 scale well, and we get sublinear returns above 5, but positive returns up to some point anyway, does this prove that AI won't FOOM until it has an intelligence larger than the largest intelligence of a group of humans? That is, until AI has a higher intelligence than the group, that the group of humans will dominate the rate at which new AI's are improved?

Comment author: CarlShulman 29 April 2013 10:50:42PM 6 points [-]
Comment author: timtyler 03 May 2013 10:56:55AM 3 points [-]

I think that teams of up to five people can scale "pretty well by human standards" - not too far from linearly. It's going up to a hundred, a thousand, a million, a billion that we start to run into incredibly sublinear returns.

That's parallelism for you. It's like the way that four-core chips are popular, while million-core chips are harder to come by.

Comment author: rhollerith_dot_com 08 May 2013 03:54:26PM *  0 points [-]

Does anyone play (rated) chess on freechess.org? If so, do you want to get together to play some team games for the purposes of adding hard data to this discussion?

My blitz rating is in the high 1200s. My teammate should have a blitz rating close to that to make the data valuable. I play 8-minute games, and am not interested in playing enough non-blitz games to get my rating to be an accurate reflection of my (individual) skill. (Non-blitz games would take too much time and take too much out of me. "Non-blitz" games are defined as games with at least 15 minutes on the clock for each player.)

I envision the team being co-located while playing, which limits my teammate to someone who is or will be in San Francisco or Berkeley.

I've played a little "team chess" before. Was a lot of fun.

My contact info is here.

Comment author: shminux 29 April 2013 08:04:20PM 18 points [-]

Some review notes as I go through it (at a bright dilettante level):

Section 1:
- I wonder if the chain-reaction model is a good one for recursive self-improvement, or is it just the scariest one? What other models have been investigated? For example, the chain-reaction model of financial investment would result in a single entity with the highest return rate dominating the Earth, this has not happened yet, to my knowledge.

Section 1.3:
- There was a recent argument here by David Pearce, I think, that an intelligent enough paperclip maximizer will have to self-modify to be more "humane". If I recall correctly, the logic was that in the process of searching the space of optimization options it will necessarily encounter an imperative against suffering or something to that effect, inevitably resulting in modifying its goal system to be more compassionate, the way humanity seems to be evolving. This would restrict the Orthogonality Thesis to the initial takeoff, and result in goal convergence later on. While this seems like wishful thinking, it might be worth addressing in some detail, beyond the footnote 11.

Chapter 2:
- log(n) + log(log(n)) + ... seems to describe well the current rate of scientific progress, at least in high-energy physics
- empty space for a meditation seems out of place in a more-or-less formal paper
- the Moore's law example is an easy target for criticism, because it's an outlier: most current technologies independent of computer progress are probably improving linearly or even sublinearly with investment (power generation, for example)
- "total papers written" seems like a silly metric to measure scientific progress, akin to the Easter Island statue size.
- If the point of the intro is to say that all types of trends happen simultaneously, and we need "to build an underlying causal model" of all trends, not cherry-pick one of them, then it is probably good to say upfront.

Section 2.1:
- the personal encounter with Kurzwell belongs perhaps in a footnote, not in the main text
- the argument that the Moore's law will speed up if it's reinvested into human cognitive speedup (isn't it now, to some degree?) assumes that faster computers is a goal in itself, I think. If there is no economic or other external reason to make faster/denser/more powerful computers, why would the cognitively improved engineers bother and who would pay them to?
- In general, the section seems quite poorly written, more like a stream of consciousness than a polished piece. It needs a decent summary upfront, at the very least. And probably a few well-structured subsections, one on the FOOM debate, one on the validity of the outside view, one on the Lucas critique, etc. It may also be worth discussing while Hanson apparently remains unconvinced.

I might add more later.

Comment author: satt 30 April 2013 12:02:17AM *  7 points [-]

- log(n) + log(log(n)) + ... seems to describe well the current rate of scientific progress, at least in high-energy physics

I'm going to commit pedantry: nesting enough logarithms eventually gives an undefined term (unless n's complex!). So where Eliezer says "the sequence log(w) + log(log(w)) + log(log(log(w))) will converge very quickly" (p. 4), that seems wrong, although I see what he's getting at.

Comment author: paper-machine 30 April 2013 12:17:17AM *  8 points [-]

It really bothers me that he calls it a sequence instead of a series (maybe he means the sequence of partial sums?), and that it's not written correctly.

The series doesn't converge because log(w) doesn't have a fixed point at zero.

It makes sense if you replace log(w) with log^+(w) = max{ log(w), 0 }, which is sometimes written as log(w) in computer science papers where the behavior on (0, 1] is irrelevant.

I suppose that amounts to assuming there's some threshold of cognitive work under which no gains in performance can be made, which seems reasonable.

Comment author: Eliezer_Yudkowsky 04 May 2013 01:29:22AM 0 points [-]

Now fixed, I hope.

Comment author: paper-machine 05 May 2013 04:22:12AM 0 points [-]

Oh yes. That makes far more sense. Thanks for fixing it.

Comment author: Eliezer_Yudkowsky 30 April 2013 06:23:39PM 1 point [-]

Since this apparently bothers people, I'll try to fix it at some point. A more faithful statement would be that we start by investing work w, get a return w2 ~ log(w), reinvest it to get a new return log(w + w2) - log(w) = log ((w+w2)/w). Even more faithful to the same spirit of later arguments would be that we have y' ~ log(y) which is going to give you basically the same growth as y' = constant, i.e., whatever rate of work output you had at the beginning, it's not going to increase significantly as a result of reinvesting all that work.

I'm not sure how to write either more faithful version so that the concept is immediately clear to the reader who does not pause to do differential equations in their head (even if simple ones).

Comment author: Vaniver 01 May 2013 12:53:14AM 0 points [-]

Well, suppose cognitive power (in the sense of amount of cognitive work put unit time) is a function of total effort invested so far, like P=1-e^(-w). Then it's obvious that while dP/dw= e^(-w) is always positive, it rapidly decreases to basically zero, and total cognitive power converges to some theoretical maximum.

Comment author: GuySrinivasan 30 April 2013 06:44:55PM 0 points [-]

This is in the context of reinvesting dividends of cognitive work, assuming it takes exponentially greater investments to produce linearly greater returns. For example, maybe we get a return of log(X) cognitive work per time with what we have now, and to get returns of log(X+k) per time we need to have invested X+k cognitive work. What does it look like to reinvest all of our dividends? After dt, we have invested X+log(X) and our new return is log(X+log(X)). After 2dt, we have invested X+log(X)+log(X+log(X)), etc.

The corrected paragraph would then look like:

Therefore, an AI trying to invest an amount of cognitive work w to improve its own performance will get returns that go as log(w), or if further reinvested, an additional log(1+log(w)/w), and the sequence log(w)+log(1+log(w)/w)+log(1+log(w+log(w))/(w+log(w))) will converge very quickly.

Except then it's not at all clear that the series converges quickly. Let's check... we could say the capital over time is f(t), with f(0)=w, and the derivative at t is f'(t)=log(f(t)). Then our capital over time is f(t)=li^(-1)(t+li(w)). This makes our capital / log-capital approximately linear, so our capital is superlinear, but not exponential.

Comment author: Vaniver 29 April 2013 08:40:26PM 7 points [-]

If I recall correctly, the logic was that in the process of searching the space of optimization options it will necessarily encounter an imperative against suffering or something to that effect, inevitably resulting in modifying its goal system to be more compassionate, the way humanity seems to be evolving.

I see no reason to suspect the space of optimization options contains value imperatives, assuming the AI is guarded against the equivalent of SQL injection attacks.

Humanity seems to be evolving towards compassion because being the causal factors increasing compassion are on average profitable for individual humans with those factors. The easy example of this is stable, strong police forces routinely hanging murderers, instead of those murderers profiting from from their actions. If you don't have an analogue of the police, then you shouldn't expect the analogue of the reduction in murders.

(I should remark that I very much like the way this report is focused; I think that trying to discuss causal models explicitly is much better than trying to make surface-level analogies.)

  • empty space for a meditation seems out of place in a more-or-less formal paper

At the very least, using a page break rather than a bunch of ellipses seems better.

Comment author: shminux 29 April 2013 11:37:13PM *  0 points [-]

Humanity seems to be evolving towards compassion because being the causal factors increasing compassion are on average profitable for individual humans with those factors.

I was simply paraphrasing David Pearce, it's not my opinion, so no point arguing with me. That said, your argument seems misdirected in another way: the imperative against suffering applies to people and animals whose welfare is not in any way beneficial and sometimes even detrimental to those exhibiting compassion.

Comment author: Jayson_Virissimo 30 April 2013 01:58:44AM *  3 points [-]

Yeah, but they are losing compassion for other things (unborn babies, gods, etc...). What reason is there to believe there is a net gain in compassion, rather than simply a shift in the things to be compassionate towards?

EDIT: This should have been directed towards Vaniver rather than shminux.

Comment author: davidpearce 17 May 2013 11:37:44AM 0 points [-]

an expanding circle of empathetic concern needn't reflect a net gain in compassion. Naively, one might imagine that e.g. vegans are more compassionate than vegetarians. But I know of no evidence this is the case. Tellingly, female vegetarians outnumber male vegetarians by around 2:1, but the ratio of male to female vegans is roughly equal. So an expanding circle may reflect our reduced tolerance of inconsistency / cognitive dissonance. Men are more likely to be utilitarian hyper-systematisers.

Comment author: Nornagest 18 May 2013 08:45:02AM 1 point [-]

Does your source distinguish between motivations for vegetarianism? It's plausible that the male:female vegetarianism rates are instead motivated by (e.g.) culture-linked diet concerns -- women adopt restricted diets of all types significantly more than men -- and that ethically motivated vegetarianism occurs at similar rates, or that self-justifying ethics tend to evolve after the fact.

Comment author: davidpearce 19 May 2013 10:40:19AM 2 points [-]

Nornagest, fair point. See too "The Brain Functional Networks Associated to Human and Animal Suffering Differ among Omnivores, Vegetarians and Vegans" : http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0010847

Comment author: Jayson_Virissimo 17 May 2013 11:23:42PM 0 points [-]

an expanding circle of empathetic concern needn't reflect a net gain in compassion. Naively, one might imagine that e.g. vegans are more compassionate than vegetarians. But I know of no evidence this is the case. Tellingly, female vegetarians outnumber male vegetarians by around 2:1, but the ratio of male to female vegans is roughly equal. So an expanding circle may reflect our reduced tolerance of inconsistency / cognitive dissonance. Men are more likely to be utilitarian hyper-systematisers.

Right. What I should have said was:

What reason is there to believe that people are compassionate towards more types of things, rather than merely different types of things?

Comment author: davidpearce 18 May 2013 08:24:05AM 1 point [-]

The growth of science has led to a decline in animism. So in one sense, our sphere of concern has narrowed. But within the sphere of sentience, I think Singer and Pinker are broadly correct. Also, utopian technology makes even the weakest forms of benevolence vastly more effective. Consider, say, vaccination. Even if, pessimistically, one doesn't foresee any net growth in empathetic concern, technology increasingly makes the costs of benevolence trivial.

[Once again, I'm not addressing here the prospect of hypothetical paperclippers - just mind-reading humans with a pain-pleasure (dis)value axis.]

Comment author: Eugine_Nier 18 May 2013 09:28:41AM 0 points [-]

But within the sphere of sentience, I think Singer and Pinker are broadly correct.

Would this be the same Singer who argues that there's nothing wrong with infanticide?

Comment author: davidpearce 18 May 2013 11:23:49AM 2 points [-]

On (indirect) utilitarian grounds, we may make a strong case that enshrining the sanctity of life in law will lead to better consequences than legalising infanticide. So I disagree with Singer here. But I'm not sure Singer's willingness to defend infanticide as (sometimes) the lesser evil is a counterexample to the broad sweep of the generalisation of the expanding circle. We're not talking about some Iron Law of Moral Progress.

Comment author: Will_Newsome 30 April 2013 06:06:26PM *  0 points [-]

I find it really weird that I don't recall having seen that piece of rhetoric before. (ETA: Argh, dangerously close to politics here. Retracting this comment.)

Comment author: Eliezer_Yudkowsky 30 April 2013 06:11:57PM 2 points [-]

I wish I could upvote your retraction.

Comment author: Adele_L 30 April 2013 07:08:19PM 8 points [-]

The closest thing I have seen to this sort of idea is this:

http://www.gwern.net/The%20Narrowing%20Circle

Comment author: Jayson_Virissimo 01 May 2013 04:50:00AM *  5 points [-]

Wow, an excellent essay!

If I remember correctly, I started thinking along these lines after hearing Robert Garland lecture on ancient Egyptian religion. As a side-note to a discussion about how they had little sympathy for the plight of slaves and those in the lower classes of society (since this was all part of the eternal cosmic order and as it should be), he mentioned that they would likely think that we are the cruel ones, since we don't even bother to feed and cloth the gods, let alone worship them (and the gods, of course, are even more important than mere humans, making our lack of concern all the more horrible).

Comment author: gwern 01 May 2013 08:01:56PM 3 points [-]

Any idea where Garland might've written that up? All the books listed in your link sound like they'd be on Greece, not Egypt.

Comment author: Jayson_Virissimo 04 May 2013 07:10:18AM 1 point [-]

It was definitely a lecture, not a book. Maybe I'll track it down when I get around to Ankifying my Ancient Egypt notes.

Comment author: Vaniver 30 April 2013 02:24:23PM 1 point [-]

it's not my opinion, so no point arguing with me.

It seems beneficial to make sure my understanding of why Pearce's argument fails matches that of others, even if I don't need to convince you that it fails.

the imperative against suffering applies to people and animals whose welfare is not in any way beneficial and sometimes even detrimental to those exhibiting compassion.

I interpret imperatives as "you should X," where the operative word is the "should," even if the content is the "X." It is not at all obvious to me why Pearce expects the "should" to be convincing to a paperclipper. That is, I don't think there is a logical argument from arbitrary premises to adopt a preference for not harming beings that can feel pain, even though the paperclipper may imagine a large number of unconvincing logical arguments whose conclusion is "don't harm beings that can feel pain if it costless to avoid" on the way to accomplishing its goals.

Comment author: davidpearce 16 May 2013 07:21:58PM *  2 points [-]

Perhaps it's worth distinguishing the Convergence vs Orthogonality theses for: 1) biological minds with a pain-pleasure (dis)value axis. 2) hypothetical paperclippers.

Unless we believe that the expanding circle of compassion is likely to contract, IMO a strong case can be made that rational agents will tend to phase out the biology of suffering in their forward light-cone. I'm assuming, controversially, that superintelligent biological posthumans will not be prey to the egocentric illusion that was fitness-enhancing on the African savannah. Hence the scientific view-from-nowhere, i.e. no arbitrarily privileged reference frames.

But what about 2? I confess I still struggle with the notion of a superintelligent paperclipper. But if we grant that such a prospect is feasible and even probable, then I agree the Orthogonality thesis is most likely true.

Comment author: Eliezer_Yudkowsky 04 May 2013 01:29:05AM 5 points [-]

For example, the chain-reaction model of financial investment would result in a single entity with the highest return rate dominating the Earth, this has not happened yet, to my knowledge.

Like... humans? Or the way that medieval moneylenders aren't around anymore, and a different type of financial organization seems to have taken over the world instead? See also the discussion of China catching up to Australia.

Comment author: shminux 05 May 2013 04:26:20AM 1 point [-]

Fair point about human domination. Though I'm not sure how it fits into the chain reaction model. Maybe reinvestment of knowledge into more knowledge does, not intelligence into more intelligence. As for financial investments, I don't know of any organization emerging as a singleton.

Comment author: RolfAndreassen 03 May 2013 04:50:40PM 17 points [-]

Agreeing with several other people that the introduction needs a major rewrite or possibly just a cut. Consider the opening sentence:

Isadore Jacob Gudak, who anglicized his name to Irving John Good and used I. J. Good for publication

Dude, no. Who gives a toss how he anglicised his name? Get to your point, if you have one.

Somewhat similarly, in the fourth paragraph, you have

Please note that...

Please note that the phrase "please note that" is unnecessary; it adds length and the impression that you are snippily correcting someone's blog comment, without adding any information (or politeness) to the sentence. I'm familiar with your argument about formal writing just adding a feeling of authority, but this isn't informality, it's sloppy editing.

Your whole first page, actually, is a pretty good demonstration of not having a point. I get the impression that you thought "Hmm, I need some kind of introduction" and went off to talk about something, anything, that wasn't the actual point of the paper, because the point belongs in the body and not the introduction. This makes for a page that adds nothing. You have a much better introduction starting with the paragraph at the end of the first page, the one that opens

The question of what happens when smarter-than-human agencies

See, this is getting to the point. You can do it! This is where you should start the paper.

At an absolute, utter minimum, move the subclause about how what's-his-name anglicised the Hungarian into a footnote, or the bibliography, or a biographical appendix, or a Wikipedia article, or for dog's sake the Author's Notes to the next HPMOR chapter, or a random thought that Harry has and then wonders why he is considering such a total irrelevancy. Just please, anywhere, anywhere except the opening sentence of your paper. Talk about burying the lede.

Additionally, your abstract is too long. The abstract should not explain anything; it should summarise the argument or result on the assumption that the reader is already familiar with the subject. You're trying to write your introduction in the abstract; a common error, but an error. A single paragraph s sufficient; going off the first page is way too long. Disclaimer: The above applies to physics abstracts; conceivably philosophy has a different set of conventions.

Finally: If in fact you actually want this kind of feedback, you can make it much easier on your beta readers by adding line numbers to the paper. LaTeX makes this easy. This avoids such circumlocutions as "The fourth paragraph, the one beginning...", with the attendant confusion about whether I'm counting the block quote as a separate paragraph.

Comment author: shminux 30 April 2013 10:55:10PM *  17 points [-]

Having looked through the document again, I feel that a competent technical writer, or anyone with a paper-writing experience, can make this report into a paper suitable for submission within a couple of days, maybe a week, assuming MIRI wants it published. A lot would have to be cut, and the rest rearranged and tidied up, but there is definitely enough meat there for a quality paper or two. I am not sure what MIRI's intention is re this report, other than "hope that the open problems posed therein inspire further work by economists or economically literate modelers".

Comment author: Vaniver 29 April 2013 06:55:21PM *  12 points [-]

Comments:

Given human researchers of constant speed, computing speeds double every 18 months.

Human researchers, using top-of-the-line computers as assistants. I get the impression this matters more for chip design than litho-tool design, but it definitely helps for those too.

Humans have around four times the brain volume of chimpanzees, but the difference between us is probably mostly software algorithms.

Is 'software algorithms' the right phrase? I'd characterize the improvements more as firmware or hardware improvements. [edit] Later you use the phrase "cognitive algorithms," which I'm much happier with.

A more concrete example you can use to replace the handwaving: one of the big programming productivity boosters is a second monitor, which seems directly related to low human working memory. It's easy to imagine minds with superior working memory able to handle much more complicated models and tasks. (We indeed seem to see this diversity among humans.)

In particular, your later arguments on serial causal depth seem like they would benefit from explicitly considering working memory as well as speed.

Any lab that shuts down overnight so its researchers can sleep must be limited by serial cause and effect in researcher brains more than serial cause and effect in instruments- researchers who could work without sleep would correspondingly speed up the lab.

I don't know about you, but I do research in my sleep, and my lab never shuts off our computers because we often have optimization processes running overnight (on every computer in the lab).

It is the case that most of the cycle time in research is mostly due to the human researchers rather than the computer speed (each month on average there might be about a week that's code-limited rather than human-limited), but this example as you present it is unconvincing.

Comment author: TheOtherDave 29 April 2013 07:51:55PM 14 points [-]

It's easy to imagine minds with superior working memory able to handle much more complicated models and tasks. [..] In particular, your later arguments on serial causal depth seem like they would benefit from explicitly considering working memory

Strong, albeit anecdotal, agreement.

Working memory capacity was a large part of what my stroke damaged, and in colloquial terms I was just stupid, relatively speaking, until that healed/retrained. I was fine when dealing with simple problems, but add even a second level of indirection and I just wasn't able to track. The effect is at least subjectively highly nonlinear.

Comment author: Vaniver 29 April 2013 08:31:15PM *  13 points [-]

Incidentally, I think this is the strongest argument against Egan's General Intelligence Theorem (or, alternatively, Deutsch's "Universal Explainer" argument from The Beginning of Infinity). Yes, humans could in theory come up with arbitrarily complex causal models, and that's sufficient to understand an arbitrarily complex causal system, but in practice, unaided humans are limited to rather simple models. Yes, we're very good at making use of aids (I'm reminded of how much writing helps thinking whenever I try to do a complicated calculation in my head), but those limitations represent a plausible way for meaningful superhuman intelligence to be possible.

Comment author: TheOtherDave 29 April 2013 08:53:00PM 16 points [-]

I hope never to forget the glorious experience of re-inventing the concept of lists, about two weeks into my recovery. I suddenly became indescribably smarter.

In the same vein, I have been patiently awaiting the development of artificial working-memory cognitive buffers. As you say, for practical purposes this is superhuman intelligence.

Comment author: Luke_A_Somers 30 April 2013 05:40:39PM 3 points [-]

Gaaah. I hate brain damage.

Congratulations on your discovery, anyway.

Comment author: TheOtherDave 30 April 2013 06:15:04PM 2 points [-]

Yeah, you and me both, brother.

Comment author: timtyler 04 May 2013 01:33:43AM *  0 points [-]

Given human researchers of constant speed, computing speeds double every 18 months.

Human researchers, using top-of-the-line computers as assistants.

Indeed. For me, that was the most glaring conceptual problem. That and attempting to predict the course of evolution with minimal reference to evolutionary theory. There is a literature on how cultural systems evolve. For a specific instance see this:

The third tipping point was the appearance of technology capable of accumulating and manipulating vast amounts of information outside humans, thus removing them as bottlenecks to a seemingly self-perpetuating process of knowledge explosion.

Comment author: somervta 30 April 2013 06:13:08AM 9 points [-]

A very succinct summary appears in Wikipedia (2013):

Citing Wikipedia in any kind of academic context is generally a bad idea, even if it's just for a summary.

Comment author: lukeprog 08 May 2013 02:36:09AM 8 points [-]

On page 15, you write:

the Moore’s-like law for serial processor speeds broke down in 2004

No citation is given, but I found one: Fuller & Millett (2011). The paper includes this handy graph:

Moore's Law graph

And also this one:

Moore's law graph #2

Comment author: lukeprog 08 May 2013 03:16:26AM 2 points [-]

The book behind the paper includes lots more detail. Its introduction is quite cheery:

The end of dramatic exponential growth in single-processor performance marks the end of the dominance of the single microprocessor in computing... There is no guarantee that we can make parallel computing as common and easy to use as yesterday’s sequential single-processor computer systems, but unless we aggressively pursue efforts suggested by the recommendations below, it will be “game over” for growth in computing performance. If parallel programming and related software efforts fail to become widespread, the development of exciting new applications that drive the computer industry will stall; if such innovation stalls, many other parts of the economy will follow suit.

Comment author: CellBioGuy 09 May 2013 02:52:15PM *  2 points [-]

I have an engineer friend who has recently put forward the idea that computing technology is approaching becoming a 'mature' technology, like the automobile in the 1950s. It gets a job done and does it well, every change made after that point is a matter of small incremental tweaks. Yeah you get twice the gas mileage now as you did then after a load of small changes with diminishing returns, but is it really all that different? Other friends of mine working as programmers have reacted favorably when I relayed this idea.

Also, why should slower development of new applications for the computer industry kill the economy?

Comment author: gjm 30 April 2013 11:41:15AM *  18 points [-]

Superficial stylistic remarks (as you'll see, I've only looked at about the first 1/4 of the paper):

  • The paper repeatedly uses the word "agency" where "agent" would seem more appropriate.

  • I agree with paper-machine that the mini-biography of I J Good has little value here.

  • The remark in section 1 about MIRI being funding-limited is out of place and looks like a whine or a plea for more money. Just take it out.

  • "albeit" on page 10, shortly before footnote 8, should just be "but". (Or maybe "even though", if that's your meaning.) [EDITED to add: there's another "albeit" that reads oddly to me, in footnote 66 on page 50. It's not wrong, but it feels odd. Roughly, wherever you can correctly write "albeit" you can equivalently write "even though", and that's a funny thing to be starting a footnote with.]

  • "criteria" in footnote 11 about paperclip maximizers should be "criterion".

  • In footnote 15 (about "g") the word "entrants" seems very weirdly chosen, and the footnote seems to define g as the observed correlation between different measures of intelligence, which is nonsense.

  • The premise of the paper is that whether or not intelligence explosion will occur is (or at least is being pretended to be) an open question. But at many points within the paper there are references to "the intelligence explosion" that seem to presuppose that there will in fact be one.

  • Footnote 23 puts "Wikipedia" in italics, which to my eyes looks very strange.

[EDITED to incorporate a comment I made separately, which I'll now retract.]

Comment author: Eliezer_Yudkowsky 04 May 2013 01:26:33AM 8 points [-]

I agree with paper-machine that the mini-biography of I J Good has little value here.

Done.

The remark in section 1 about MIRI being funding-limited is out of place and looks like a whine or a plea for more money. Just take it out.

Done.

Comment author: Qiaochu_Yuan 30 April 2013 09:36:04AM *  15 points [-]
  • Page 4: the sum log(w) + log(log(w)) + ... doesn't converge. Some logarithm will be negative and then the next one will be undefined. Presumably you meant to stop the sum once it becomes negative, but then I'm somewhat confused about this argument because I'm not sure it's dimensionally consistent (I'm not sure what units cognitive work is being measured in).

  • Top of page 18: there's a reference to "this graph" but no graph...?

General comment 1: who's the intended audience here? Most of the paper reads like a blog post, which I imagine could be disconcerting for newcomers trying to evaluate whether they should be paying attention to MIRI and expecting a typical research paper from a fancy-looking .pdf.

General comment 2: I still think this discussion needs more computational complexity. I brought this up to you earlier and I didn't really digest your reply. The question of what you can and can't do with a given amount of computational resources seems highly relevant to understanding what the intelligence explosion could look like; in particular I would be surprised if questions like P vs. NP didn't have a strong bearing on the distribution over timelines (I expect that the faster it is possible to solve NP-complete problems, which apparently includes protein folding, the faster AI could go foom). But then I'm not a domain expert here and I could be off-base for various reasons.

Comment author: lukeprog 30 April 2013 04:31:27PM 3 points [-]

I still think this discussion needs more computational complexity

IIRC, Daniel Dewey said at the April workshop that this is, in fact, his current project.

Comment author: Eliezer_Yudkowsky 30 April 2013 02:23:14PM 0 points [-]

Protein folding cannot be NP-hard. The physical universe is not known to be able to solve NP-hard problems, and protein folding will not involve new physics.

Comment author: Kindly 30 April 2013 09:27:37PM 18 points [-]

The physical universe doesn't need to "solve" protein folding in the sense of having a worst-case polynomial-time algorithm. It just needs to fold proteins. Many NP-complete problems are "mostly easy" with a few hard instances that rarely come up. (In fact, it's hard to find an NP-complete problem for which random instances are hard: if we could do this, we would use it for cryptography.) It's reasonable to suppose protein folding is like this.

Of course, if this is the case, maybe the AI doesn't care about the rare hard instances of protein folding, either.

Comment author: EHeller 30 April 2013 07:17:03PM *  4 points [-]

Proteins are finite in length. Why would nature care if it can't do something in polynomial time?

Edit: It would be interested to turn this around- suppose proteins folding IS NP-hard, can we put an upper bound on the length of proteins using evolutionary time scales?

Comment author: CellBioGuy 09 May 2013 03:02:55PM *  7 points [-]

can we put an upper bound on the length of proteins using evolutionary time scales?

Not really, most big proteins consist of 'domains' which fold up pretty independantly of each other (smaller proteins can vary quite a bit though). Titin is a ~30,000 amino acid protein in human muscle with ~500 repeats of the same 3 basic modules all laid out in a line... over evolutionary time you can shuffle these functional units around and make all kinds of interesting combinations.

Actually, the lab I'm working in recently had a problem with this. We optimized a gene to be read extremely fast by a ribosome while still producing exactly the same protein sequence (manipulating synonymous codons). But it turned out that when you have the actual protien molecule being extruded from the ribosome as rapidly as we had induced it to be, the normal independant folding of successive domains was disrupted - one domain didn't fully fold before the next domain started being extruded, they interacted, and the protein folded all wrong and didn't work despite having exactly the same sequence as the wild protein.

Comment author: Qiaochu_Yuan 30 April 2013 07:25:21PM 7 points [-]

I think the more important point is that Nature doesn't care about the worst case (if a protein takes forever to fold correctly then it's not going to be of any use). But an AI trying to design arbitrary proteins plausibly might.

Comment author: Eliezer_Yudkowsky 30 April 2013 11:45:05PM 3 points [-]

Why? Nature designed ribosomes without solving any hard cases of NP-hard problems. Why would Ribosomes 2.0, as used for constructing the next generation of molecular machinery, require any NP-hard problems?

Comment author: [deleted] 01 May 2013 02:04:42AM *  5 points [-]

Nature designed ribosomes without solving any hard cases of NP-hard problems.

Is that so? Pretty much every problem of interest (chess, engineering, etc) is NP-hard, or something on that level of difficulty.

The thing with such problems is that you don't solve them exactly for optimal, you weaken the problem and solve them heuristically for good. Nature did just this: produced a solution to a weakened NP-hard design problem.

I think at this point, taboo "solved a hard problem". Nature produced a better replicator, not necessarily the optimal superbest replicator, which it didn't have to.

Obviously an intelligently designed automated design system could be just as good as dumbass evolution, given sufficient computing power, so I agree that swiftly designing nanomachinery is quite plausible.

(The advantage that evolution has over an engineer in a leaky box with a high discount rate is that there's a lot going on in molecular dynamics, solutions are dense in the space, as shown by that evolution got some, but that's no guarantee that a given solution is predictably a solution. So it might be cost prohibitive. (Not that I'm up to date on protein folding.)

In the worst case, the protein folding could be a secure hash, that takes a very expensive high-fidelity simulation to compute. Evolution would do just fine at cracking hashes by brute force, because it bruteforces everything, but an intelligent engineer wouldn't be able to do much better in this case. It may end up taking too much computation to run such sims. (I should calculate the expected time to nano in this case, it would be a very interesting fermi estimate).

However, it is highly unlikely for lower-fidelity simulations and models to give you literally no information, which would be what is required for the "secure hash" scenario. Even things designed to be secure hashes often lose huge parts of their theoretical complexity after some merely human attacks (eg md5). The mainline case is that there exist reasonable heuristics discoverable by reasonable amounts of intelligent computation. (likewise should fermi here, but would need some harder assumptions)

Of course you already knew all this... (But I didn't))

So yeah, this just made nano seem a bit more real to me.

Comment author: Eliezer_Yudkowsky 01 May 2013 02:20:37AM 8 points [-]

In the worst case, the protein folding could be a secure hash

Then it would be harder, in fact impossible, to end up with slightly better proteins via point mutations. A point mutation in a string gives you a completely different secure hash of that string.

This isn't a minor quibble, it's a major reason to go "Whaa?" at the idea that protein folding and protein design have intractable search spaces in practice. They have highly regular search spaces in practice or evolution couldn't traverse that space at all.

Comment author: [deleted] 01 May 2013 03:10:02AM *  4 points [-]

This isn't a minor quibble, it's a major reason to go "Whaa?" at the idea that protein folding and protein design have intractable search spaces in practice. They have highly regular search spaces in practice or evolution couldn't traverse that space at all.

Yep, it's highly implausible that a natural non-designed process would happen to be a secure hash (as you know from arguing with cryonics skeptics). And that's before we look at how evolution works.

Good point. The search space is at least smooth once in a few thousand tries. (While doing the nearby fermi estimate, I saw a result that 12% (!!!) of point mutations in some bacteria were beneficial).

That said, the "worst possible case" is usually interesting.

Comment author: DanArmak 02 May 2013 04:47:16PM 4 points [-]

Pretty much every problem of interest (chess, engineering, etc) is NP-hard, or something on that level of difficulty.

Isn't that in large part a selection effect? After decades having computers, most of the low hanging fruit has been picked, and so many unsolved problems are NP-hard. But many equally important problems have been solved because they weren't.

Comment author: [deleted] 01 May 2013 03:03:02AM *  4 points [-]

(I should calculate the expected time to nano in this case, it would be a very interesting fermi estimate).

Lets go!

From Wik:

In early 2008, Rosetta was used to computationally design a protein with a function never before observed in nature.

Skimmed the paper looks like they used the rosetta@home network (~9 TFLOPS) to design a rudimentary enzyme.

So that suggests that a small amount of computation (bearable time by human research standards, allowing for fuckups and restarts) can do protein design. Let's call it a week of computation total. There's 1e6 seconds in a week, flopping at a rate of 1e13 flops, giving us 1e19 flops.

They claimed to have tested 1e18 somethings, so our number is plausible, but we should go to at least 1e22 flops to include 1e4 flops per whatever. (which would take a thousand weeks?) something doesn't add up. Whatever, call it 1e20 (ten weeks) and put some fat error bounds on that.

Don't know how to deal with the exponential complexity. A proper nanothing could require 1e40 flops (square the exponent for double complexity), or it may factor nicely, requiring only 1e21 flops.

Let's call it 1e25 flops with current techniques to design nanotech.

If AI is in 20 years, that's 13 moores doublings or 1e4, then let's say the AI can seize a network of as much computational power as they used, plus moores scaling.

So 1e21 todayflops, 1e20 of which is doable in a standard research project amount of time with a large distributed network.

So anywhere from days to 20 years, with my numbers giving 2 years, to brute force nanotech on 20-years-in-future computational power with today's algorithms.

Factor of 1e6 speedups are reasonable in chess (another problem with similar properties) with a bunch of years of human research, so that puts my middle at 10 minutes.

The AI will probably do better than that, but that would be good enough to fuck us.

This was somewhat conservative, even. (nanotech involves 100000 times more computation than these guys used)

Let's get this thing right the first time....

EDIT: an interesting property of exponential processes is that things go from "totally impossible" to "trivial" very quickly.

Comment author: [deleted] 01 May 2013 04:14:47AM 2 points [-]

Note that by these estimates, humans should be able to have nano around 2030. Scary stuff.

Comment author: Qiaochu_Yuan 01 May 2013 01:21:08AM 3 points [-]

I can feel some inferential distance here that isn't successfully being bridged. It's far from clear to me that the default assumption here should be that no NP-hard problems need to be solved and that the burden of proof is on those who claim otherwise.

Comment author: Eliezer_Yudkowsky 01 May 2013 02:17:17AM 0 points [-]

I guess to me the notion of "solve an NP-hard problem" (for large N and hard cases, i.e., the problem really is NP hard) seems extremely exotic - all known intelligence, all known protein folding, and all known physical phenomena must be proceeding without it - so I feel a bit at a loss to relate to the question. It's like bringing up PSPACE completeness - I feel a sense of 'where did that come from?' and find it hard to think of what to say, except for "Nothing that's happened so far could've been PSPACE complete."

Comment author: Kindly 02 May 2013 02:14:53AM 2 points [-]

Agreed if you mean "Nothing that's happened so far could've been [computationally hard] to predict given the initial conditions."

But the reverse problem -- finding initial conditions that produce a desired output -- could be very hard. Nature doesn't care about this, but an AI plausibly might.

I'm not sure how protein folding fits into this picture, to be honest. (Are people just trying to figure out what happens to a given protein in physics, or trying to find a protein that will make something good happen?) But more generally, the statement "P=NP" is more or less equivalent to "The reverse problem I mention above is always easy." Things become very different if this is true.

Comment author: Vaniver 30 April 2013 02:35:00PM 4 points [-]

Protein folding cannot be NP-hard. The physical universe is not known to be able to solve NP-hard problems, and protein folding will not involve new physics.

Here's a paper claiming that the most widely studied model of protein folding in 1998 is NP-Complete. I don't know enough about modern research into protein folding to comment how applicable that result still is.

My guess is you're referring to Aaronson's paper, which doesn't seem relevant here. The universe doesn't solve NP-hard problems in P time, but the universe took NP time to build the first useful proteins, didn't it?

Comment author: zslastman 30 April 2013 03:34:33PM *  15 points [-]

The solution space is large enough that even proteins sampling it's points at a rate of trillions per second couldn't really fold if they were just searching randomly through all possible configurations, that would be NP complete. They don't actually do this of course. Instead they fold piece by piece as they are produced, with local interactions forming domains which tend to retain their approximate structure once they come together to form a whole protein. They don't enter the lowest possible energy state therefore. Prion diseases are an example of what can happen when proteins enter a normally inaccessible local energy minimum, which in that case happens to have a snowballing effect on other proteins.

The result is that they follow grooves in the energy landscape towards an energy well which is robust enough to withstand all sorts of variation, including the horrific inaccuracies of our attempts at modeling. Our energy functions are just very crude approximations to the real one, which is dependent on quantum level effects and therefore intractable. Another issue is that proteins don't fold in isolation - they interact with chaperone proteins and all sorts of other crap. So simulating their folding might require one to simulate a LOT of other things besides just the protein in question.

Even our ridiculously poor attempts at in silico folding are not completely useless though. They can even be improved with the help of the human brain (see Foldit). I think an A.I. should make good progress on the proteins that exist. Even if it can't design arbitrary new ones from scratch, intelligent modification of existing ones would likely be enough to get almost anything done. Note also that an A.I. with that much power wouldn't be limited to what already exists, technology is already in the works to produce arbitrary non-protein polymers using ribosome like systems, stuff like that would open up an unimaginably large space of solutions that existing biology doesn't have access to.

Comment author: Vaniver 30 April 2013 03:44:44PM 4 points [-]

Right, this gets called Levinthal's Paradox.

Comment author: zslastman 30 April 2013 03:51:34PM *  4 points [-]

Right. I'm actually not sure how relevant it all is to discussions of an A.I. trying to get arbitrary things done with proteins. Folding an existing protein may be a great deal easier than finding a protein which folds into an arbitrary shape. Probably not all shapes are allowed by the physics of the problem. Evolution can't really be said to solve that problem either. It just produces small increments in fitness. Otherwise organisms' proteomes would be a lot more efficient.

Although on second thought, an A.I. would probably just be able to design a protein with a massively low energy funnel, so that even if it couldn't simulate folding perfectly, it could still get things done.

Regardless, an imperfect solution would probably suffice for world domination...

Comment author: Eliezer_Yudkowsky 30 April 2013 03:10:08PM 4 points [-]

I believe I've seen that discussed before and the answer is just that in real life, proteins don't fold into the lowest-energy conformation. It's like saying that the global minimum energy for soap bubbles is NP-complete. Finding new useful proteins tends to occur via point mutations so that can't be NP-complete either.

Comment author: Vaniver 30 April 2013 03:35:04PM *  4 points [-]

So, I can think of several different things that could all be the 'protein folding problem.'

  1. Figure out the trajectory an unfolded protein takes towards a folded protein, with a known starting state. (P)
  2. Given a known folded protein, find local minima that unfolded proteins starting with random start states might get stuck in. (NP, probably)
  3. Given a desired chemical reaction, find a folded protein that will catalyze the reaction. (Not sure, probably NP.)
  4. Given a desired chemical reaction, find a folded protein that will catalyze the reaction that is the local minimum reached by most arbitrary unfolded positions for that protein (Optimal is definitely NP, but I suspect feasible is too.)
  5. Others. (Here, the closest I get to molecular nanotech is 'catalyze reactions,' but I imagine the space for 'build a protein that looks like X' might actually be smaller.)

It looks to me like the problems here that have significant returns are NP.

Finding new useful proteins tends to occur via point mutations so that can't be NP-complete either.

It's not at all clear to me what you mean by this. I mean, take the traveling salesman problem. It's NP-Hard*, but you can get decent solutions by using genetic algorithms to breed solutions given feasible initial solutions. Most improvements to the route will be introduced by mutations, and yet the problem is still NP-hard.

That is, it's not clear to me that you're differentiating between the problem of finding an optimal solution being NP hard, it taking NP time to find a 'decent' solution, and an algorithm requiring NP time to finish running.

(The second is rarely true for things like the traveling salesman problem, but is often true for practical problems where you throw in tons of constraints.)

* A variant is NP-Complete, which is what I originally wrote.

Comment author: Eliezer_Yudkowsky 30 April 2013 04:08:04PM 0 points [-]

Nothing that has physically happened on Earth in real life, such as proteins folding inside a cell, or the evolution of new enzymes, or hominid brains solving problems, or whatever, can have been NP-hard. Period. It could be a physical event that you choose to regard as a P-approximation to a theoretical problem whose optimal solution would be NP-hard, but so what, that wouldn't have anything to do with what physically happened. It would take unknown, exotic physics to have anything NP-hard physically happen. Anything that could not plausibly have involved black holes rotating at half the speed of light to produce closed timelike curves, or whatever, cannot have plausibly involved NP-hard problems. NP-hard = "did not physically happen". "Physically happened" = not NP-hard.

Comment author: Tyrrell_McAllister 30 April 2013 06:42:22PM *  17 points [-]

Nothing that has physically happened on Earth in real life, such as proteins folding inside a cell, or the evolution of new enzymes, or hominid brains solving problems, or whatever, can have been NP-hard. Period.

I've seen you say this a couple of times, and your interlocutors seem to understand you, even when they dispute your conclusion. But my brain keeps returning an error when I try to parse your claim.

Read literally, "NP-hard" is not a predicate that can be meaningfully applied to individual events. So, in that sense, trivially, nothing that happens (physically or otherwise, if "physically" is doing any work here) can be NP-hard. But you are evidently not making such a trivial claim.

So, what would it look like if the physical universe "solved an NP-hard problem"? Presumably it wouldn't just mean that some actual salesman found a why to use existing airline routes to visit a bunch of pre-specified cities without revisiting any one of them. Presumably it wouldn't just mean that someone built a computer that implements a brute-force exhaustive search for a solution to the traveling salesman problem given an arbitrary graph (a search that the computer will never finish before the heat death of the universe if the example is large). But I can't think of any other interpretation to give to your claim.

Comment author: CarlShulman 30 April 2013 10:46:22PM *  5 points [-]

ETA: this is a side point.

Here's Scott Aaronson describing people (university professors in computer science and cognitive science at RPI) who claim that the physical universe efficiently solves NP-hard problems:

It was only a matter of time before someone put the pieces together. Last summer Bringsjord and Taylor [24] posted a paper entitled “P=NP” to the arXiv. This paper argues that, since (1) finding a Steiner tree is NP-hard, (2) soap bubbles find a Steiner tree in polynomial time, (3) soap bubbles are classical objects, and (4) classical physics can be simulated by a Turing machine with polynomial slowdown, it follows that P = NP.

My immediate reaction was that the paper was a parody. However, a visit to Bringsjord’s home page2 suggested that it was not. Impelled, perhaps, by the same sort of curiosity that causes people to watch reality TV shows, I checked the discussion of this paper on the comp.theory newsgroup to see if anyone recognized the obvious error. And indeed, several posters pointed out that, although soap bubbles might reach a minimum-energy configuration with a small number of pegs, there is no “magical” reason why this should be true in general. By analogy, a rock in a mountain crevice could reach a lower-energy configuration by rolling up first and then down, but it is not observed to do so.

In other news, Bringsjord also claims to show by a modal argument, similar to the theistic modal argument (which he also endorses), that human brains are capable of hypercomputation: "it's possible humans are capable of hypercomputation, so they are capable of hypercomputation." For this reason he argues that superhumanly intelligent Turing machines/Von Neumann computers are impossible and belief in their possibility is fideistic.

Comment author: EHeller 30 April 2013 10:55:00PM *  6 points [-]

Here's Scott Aaronson describing people (university professors at RPI) who claim that the physical universe efficiently solves NP-hard problems.

This doesn't refute what you are responding to. Saying the universe can't solve a general NP problem in polynomial time is not the same thing as saying the universe cannot possibly solve specific instances of generally NP-complete problems, which is Tyrrell_McAllister's point, as far as I can parse. In general, the traveling salesman is NP-complete, however there are lots of cases where heuristics get the job done in polynomial time, even if those heuristics would run-away if they were given the wrong case.

To use Aaronson's soap bubbles, sometimes the soap bubble finds a Steiner tree, sometimes it doesn't. When it DOES, it has solved one instance of an NP-complete problem fairly quickly.

Comment author: Vaniver 30 April 2013 11:14:22PM 3 points [-]

But my brain keeps returning an error when I try to parse your claim.

I agree with your parse error. It looks like EY has moved away from the claim made in the grandparent, though.

Comment author: Luke_A_Somers 30 April 2013 05:45:58PM 11 points [-]

That seems a little strongly put - NP-hard scales very poorly, so no real process can take N up to large numbers. I can solve the traveling salesman problem in my head with only modest effort if there are only 4 stops. And it's trivial if there are 2 or 3 stops.

Comment author: Eliezer_Yudkowsky 30 April 2013 06:12:51PM 8 points [-]

Conceded.

Comment author: CronoDAS 15 July 2013 03:59:21AM 1 point [-]

Um... doesn't it take exponential time in order to simulate quantum mechanics on a classical computer?

Comment author: pengvado 17 July 2013 04:01:09AM *  0 points [-]

Yes (At least that's the general consensus among complexity theorists, though it hasn't been proved.) This doesn't contradict anything Eliezer said in the grandparent. The following are all consensus-but-not-proved:

P⊂BQP⊂EXP
P⊂NP⊂EXP
BQP≠NP (Neither is confidently predicted to be a subset of the other, though BQP⊂NP is at least plausible, while NP⊆BQP is not.)
If you don't measure any distinctions finer than P vs EXP, then you're using a ridiculously coarse scale. There are lots of complexity classes strictly between P and EXP, defined by limiting resources other than time-on-a-classical-computer. Some of them are tractable under our physics and some aren't.

Comment author: bogdanb 16 July 2013 02:57:13AM 0 points [-]

Is that just that we don’t know any better algorithms, or is there a proof that exptime is needed?

Comment author: CronoDAS 16 July 2013 03:33:36AM 1 point [-]

I really don't know; some Wikipedia browsing suggests that there's a proof, but I'd rather have a statement from someone who actually knows.

Comment author: JoshuaZ 30 April 2013 03:27:44PM 1 point [-]

Finding new useful proteins tends to occur via point mutations so that can't be NP-complete either.

This does not follow. It may be that finding new useful proteins just takes a very long time and is very inefficient. The rest of your comment seems correct though.

Comment author: Eliezer_Yudkowsky 30 April 2013 04:02:40PM 2 points [-]

Then evolution wouldn't happen in real life.

Actually, even that understates the argument. If you can take a 20,000 base sequence and get something useful by point-perturbing at least one of the 20,000 bases in the sequence, then whatever just happened was 50 lightyears from being NP-hard - you only had to search through 19 variants on each of 20,000 cases.

Comment author: JoshuaZ 30 April 2013 07:00:06PM 0 points [-]

Huh? How does this argument work? That doesn't mean that evolution can't happen in real life, it would be a reason to think that evolution is very slow (which it is!) or that evolution is missing a lot of interesting proteins (which seems plausible).

If you can take a 20,000 base sequence and get something useful by point-perturbing at least one of the 20,000 bases in the sequence, then whatever just happened was 50 lightyears from being NP-hard - you only had to search through 19 variants on each of 20,000 cases.

I'm not sure I follow your logic. Are you arguing that because log 20,000 <19? Yes, you can check every possible position in a base sequence this way, but there are still a lot more proteins than those 19. One doesn't get something special from just changing a specific base. Moreover, even if something interesting does happen for changing a specific one, it might not happen if one changes some other one.

Comment author: CarlShulman 30 April 2013 11:12:33PM *  3 points [-]

missing a lot of interesting proteins (which seems plausible).

Definitely, since evolution keeps introducing new interesting proteins.

That doesn't mean that evolution can't happen in real life, it would be a reason to think that evolution is very slow (which it is!)

But it's not slow on a scale of e^n for even modestly large n. If you can produce millions of proteins with hundreds to thousands of amino acids in a few billion years, then approximate search for useful proteins is not inefficient like finding the lowest-energy conformation is (maybe polynomial approximation, or the base is much better, or functional chunking lets you effectively reduce n greatly...).

Comment author: SilasBarta 01 May 2013 12:23:25AM *  0 points [-]

[...evolution is] missing a lot of interesting proteins (which seems plausible).

Definitely, since evolution keeps introducing new interesting proteins.

Wait, the fact the evolution is often introducing a interesting new proteins is evidence that evolution is missing a lot of interesting proteins? How does that follow?

Switch the scenario around: if evolution never produced interesting new proteins (anymore, after time T), would that be evidence that there are no other interesting proteins than what evolution produced?

Comment author: CarlShulman 01 May 2013 12:35:41AM *  5 points [-]

Switch the scenario around: if evolution never produced interesting new proteins (anymore, after time T), would that be evidence that there are no other interesting proteins than what evolution produced?

Yes.

That would be evidence that the supply of interesting proteins had been exhausted, just as computer performance at tic-tac-toe and checkers has stopped improving. I don't see where you're coming from here.

Comment author: Kawoomba 30 April 2013 03:33:02PM *  0 points [-]

It occurs to me that we can't really say, since we only have access to the time of the program, which may or may not reflect the actual computational resources expended.

Imagine you were living in a game, and trying to judge the game's hardware requirements. If you did that by looking at a clock in the game, you'd need to assume that the clock is synchronized to the actual system time. If you had a counter you increased, you wouldn't be able to say from inside the program every which step you get to that counter++ instruction.

The problem being that we don't have access to anything external, we aren't watching the Turing Machine compute, we are inside the Turing Machine "watching" the effects of other parts of the program, such as a folding protein (observing whenever it's our turn to be simulated). We don't, however, see the Turing Machine compute, we only see the output. The raw computing power / requirements "behind the scenes", even if such a behind the scenes is only a non-existent abstraction, is impossible to judge with certainty, similar to a map-territory divide. Since there is no access in principle, we cannot observe anything but the "output", we have no way of verifying any assumptions about a correspondence between "game timer" and "system timer" we may make, or of devising any experiments.

Even the recently linked "test the computational limits" doesn't break the barrier, since for all we know the program may stall, and the next "frame" it outputs may still seem consistent, with no stalling, when viewed from inside the program, which we are. We wouldn't subjectively realize the stall. If such an experiment did find something, it would be akin to a bug, not to a measurement of computational resources expended.

Back to diapers.

Comment author: RichardKennaway 30 April 2013 07:26:06PM 1 point [-]

It occurs to me that we can't really say, since we only have access to the time of the program, which may or may not reflect the actual computational resources expended.

That's a valid point, but it does presuppose exotic new physics to make that substrate, in which "our" time passes arbitrarily slowly compared to the really real time, so that it can solve NP-hard problems between our clock ticks. We would, in effect be in a simulation. Evidence of NP-hard problems actually being solved in P could be taken as evidence that we are in one.

Comment author: ESRogs 30 April 2013 06:44:11PM 1 point [-]

we only have access to the time of the program [...] we are inside the Turing Machine "watching" the effects of other parts of the program, such as a folding protein

If we assume that protein folding occurs according to the laws of quantum mechanics, then it shouldn't tell us anything about the computational complexity of our universe besides what quantum mechanics tells us, right?

Comment author: Kawoomba 30 April 2013 07:02:42PM *  1 point [-]

Well, yea that's what I'm leaning towards. The laws of physics themselves need not govern the machine (Turing or otherwise), they are effects we observe, us being other effects. The laws of physics and the observers both are part of the output.

Like playing an online roleplaying game and inferring what the program can actually do or what resources it takes, when all you can access is "how high can my character jump" and other in-game rules. The rules regarding the jumping, and any limits the program chose to confer to the jumping behavior are not indicative of the resource requirements and efficiency of the underlying system. Is calculating the jumping easy or hard for the computer? How would you know as a character? The output, again, is a bad judge, take this example:

Imagine using an old Intel 386 system which you rigged into running the latest FPS shooter. It may only output one frame every few hours, but as a sentient character inside that game you wouldn't notice. Things would be "smooth" for you because the rules would be unchanged from your point of view.

We can only say that given our knowledge of the laws of physics, the TM running the universe doesn't output anything which seems like an efficient NP-problem solver, whether the program contains one, or the correct hardware abstraction running it uses one, is anyone's guess. (The "contains one" probably isn't anyone's guess because of Occam's Razor considerations.)

If this is all confused (it may well be, was mostly a stray thought), I'd appreciate a refutation.

Comment author: ESRogs 01 May 2013 08:38:50AM 3 points [-]

If I understand correctly you're saying that what is efficiently computable within a universe is not necessarily the same as what is efficiently computable on a computer simulating that universe. That is a good point.

Comment author: Kawoomba 01 May 2013 08:51:13AM 0 points [-]

Exactly. Thanks for succinctly expressing my point better than I could.

The question is whether assuming a correspondence as a somewhat default case (implied by the "not necessarily") is even a good default assumption.

Why would the rules inherent in what we see inside the universe be any more indicative of the rules of the computer simulating that universe than the rules inside a computer game are reflective of the instruction set of the CPU running it (they are not)?

I am aware that the reference class "computer running super mario brother / kirby's dream land" implies for the rules to be different, but on what basis would we choose any reference class which implies a correspondence?

Also, I'm not advocating simulationism with this per se, the "outer" computer can be strictly an abstraction.

Comment author: Qiaochu_Yuan 30 April 2013 09:09:49PM 0 points [-]

Is this your complete response? I guess I could expand this to "I expect all the problems an AI needs to solve on the way to an intelligence explosion to be easy in principle but hard in practice," and I guess I could expand your other comments to "the problem sizes an AI will need to deal with are small enough that asymptotic statements about difficulty won't come into play." Both of these claims seem like they require justification.

Comment author: Eliezer_Yudkowsky 30 April 2013 09:16:33PM -1 points [-]

It's not meant as a response to everything, just noting that protein structure prediction can't be NP-hard. More generally, I tend to take P!=NP as a background assumption; I can't say I've worried too much about how the universe would look different if P=NP. I never thought superintelligences could solve NP-hard problems to begin with, since they're made out of wavefunction and quantum mechanics can't do that. My model of an intelligence explosion just doesn't include anyone trying to do anything NP-hard at any point, unless it's in the trivial sense of doing it for N=20 or something. Since I already expect things to local FOOM with P!=NP, adding P=NP doesn't seem to change much, even if the polynomial itself is small. Though Scott Aaronson seems to think there'd be long-term fun-theoretic problems because it would make so many challenges uninteresting. :)

Comment author: gjm 30 April 2013 01:56:57PM 0 points [-]

The reference to "this graph" is a hyperlink. There are many such hyperlinks in the document. They feel rather weird, and are easy to miss, given the generally print-like typesetting. It might be worth writing them all out in something like the form one would use in an ordinary printed document, while preserving their hyperlinkiness.

Comment author: shminux 01 May 2013 04:36:15AM 5 points [-]

I'm also wondering about the estimated FOOM date of 2035 (presumably give or take a decade), is there an explicit calculation of it, and hopefully the confidence intervals as well?

Comment author: Eliezer_Yudkowsky 01 May 2013 04:26:29PM 0 points [-]

Where does it say 2035 in the text? How did you get the impression that this was an estimation?

Comment author: shminux 01 May 2013 04:30:48PM 6 points [-]

Maybe I misunderstood this passage:

(Even this kind of “I don’t know” still has to correspond to some probability distribution over decades, just not a tight distribution. I’m currently trying to sort out with Carl Shulman why my median is more like 2035 and his median is more like 2080. Neither of us thinks we can time it down to the decade—we have very broad credible intervals in both cases—but the discrepancy between our “I don’t knows” is too large to ignore.)

Comment author: Eliezer_Yudkowsky 01 May 2013 04:40:22PM 3 points [-]

Hm. Thanks for pointing that out. Maybe I should remove the specific dates from there and just say we were 45 years apart. I think in a lot of ways trying to time the intelligence explosion is a huge distraction. An important probability distribution, but still a huge distraction.

Comment author: shminux 01 May 2013 04:50:38PM *  13 points [-]

Well, you mentioned on occasion that this date affects your resource allocation between CFAR and MIRI, so it might be a worthwhile exercise to make the calculation explicit and subject to scrutiny, if not in the report, then some place else.

Comment author: Eliezer_Yudkowsky 01 May 2013 05:42:07PM 6 points [-]

Fair point. We're still struggling to express things verbally, but yeah.

Comment author: shminux 29 April 2013 11:47:55PM *  5 points [-]

Section 5:
The initial effort to get some numerical models going could be overestimated, unless such models have been done already. At the very least, a small-scale effort can pin-point the hard issues. This reminds me of the core-collapse Supernova modeling: it was reasonably easy to get the explosion modeled, except for the ignition by the initial shock wave. We still don't know what exactly makes them go FOOM. Most models predict a fizzle instead of an explosion. This is likely just a surface analogy, but it might well be that a few months of summer student-level simulations, as opposed to a few years of a PhD-level work, would point to the weak links in the model.

Comment author: gjm 30 April 2013 02:04:27PM 13 points [-]

The discussion of the Yudkowsky-Hanson debate feels rather out of place. The points made are mostly highly relevant to the paper; the fact that they were made during an online debate is less so; the particular language used by either side in that debate still less. This discussion is also particularly informal and blog-post-like (random example: footnote 30), which may or may not be a problem depended on the intended audience for the paper.

I'd recommend major reworking of this section, still addressing the same issues but no longer so concerned with what each party said, or thought, or was happy to concede during that particular debate.

Comment author: shminux 01 May 2013 08:55:18PM *  4 points [-]

Here is my takeout from the report. It is not a summary, and some of the implications are mine.

The 4 Theses (conjectures, really):

  1. Inevitability of Intelligence (defined as cross-domain optimization power) Explosion due to recursive self-improvement
  2. Orthogonality (intelligence/rationality is preference-independent)
  3. Instrumental Convergence (most optimizers compete for the same resources)
  4. Complexity of Value (values are not easily formalizable, no 3 laws of robotics)
    if true, imply that AGI is an x-risk, because an AGI emerging in an ad hoc fashion will compete with humans and inevitably win.

The difference between 1-3 and 4 is that 1-3 are outside of human control, but there is a hope for solving 4, hence FAI research.

There are a few outs, which the report considers unlikely:

  • If Intelligence Explosion is not inevitable, preventing it may avert the x-risk. If it's impossible, we are in the clear, as well.
  • If all sufficiently advanced agents tend to converge on "humane" values, a la David Pearce, we have nothing to worry about
  • If powerful optimizers are likely to find their own resources, they might leave us alone

Given the above, the obvious first step is formalizing each of the theses 1-3 as a step toward evaluating their validity. The report outlines potential steps toward formalizing thesis 1, Intelligence Explosion (IE):

  • Step 1 is basically categorizing existing positions on IE and constructing explicit models for them, hopefully somewhat formal, checking them for self-consistency and comparing them with past precedents, where possible
  • Step 2 is comparing the models formalized in Step 1 by constructing a common theory of which they would be sub-cases (I am not at all sure if this is what Eliezer means)
  • Step 3 is constructing a model which is likely to contain "reality" and eventually be able to answer the question "AI go FOOM?" with some probability it is confident in.

The answer to this last question would then determine the direction of the FAI effort, if any.

This was basically the content of the first 4 chapters, as far as I can tell. (Chapter 2 is the advocacy of an outside view and chapter 3 is mostly advocacy of the hard take-off.) Chapters 5 and 6 are a mix of open questions in IE relevant to the Step 3 above, some speculations and MIRI policy arguments, as well as some musings about the scope/effort/qualifications required to tackle the problem.

Comment author: Eliezer_Yudkowsky 29 April 2013 06:25:34PM 4 points [-]

For lack of any huge problems discovered so far, moving this to Main.

Comment author: gwern 29 April 2013 09:57:02PM 13 points [-]

It's 40,000 words, you say. How fast exactly do you expect any huge problems to be found?

Comment author: Eliezer_Yudkowsky 30 April 2013 12:23:02AM 13 points [-]

If it's a huge problem like "I can't download" or "all these pages are blank" then relatively fast.

Comment author: gjm 30 April 2013 04:40:04PM 10 points [-]

I am unconvinced by the argument that H. sapiens can't be right at a limit on brain size because some people have larger-than-average heads without their mothers being dead.

Presumably head size is partly determined by environmental factors outside genetic control, and presumably having your mother die in childbirth is a really big disadvantage, much worse than being slightly less intelligent. If that's so, then what should it look like if we, as a species, are hard against that wall? (Which I take to mean that any overall increase in head size would be bad even if being cleverer is a big advantage.) I suggest we'd see head sizes that are far enough away from outright disaster that, even given that random environmental variation, death in childbirth is still pretty rare, but not completely unknown. And, of course, that's just what we see; death in childbirth is very rare now, in prosperous advanced countries, but if you go back 100 years or look in less fortunate parts of the world it's not so rare at all.

This could be quantified, at least kinda. We could look at how the frequency of death in childbirth, in places without modern medical care, varies with head size (though controlling this adequately would be hard); we could look at the (rather weak and confounded with other things) relationship between head size and intelligence; and we could say that it looks as if d(IQ)/d(head size) points of IQ are about as valuable, evolutionarily, as a probability -d(maternal mortality)/d(head size) of having your mother die when you're born.

Without doing that calculation or something like it, I don't think Eliezer is justified in saying that it doesn't look as if there's been strong selective pressure for bigger brains; it seems to me like the pressure could be pretty strong without the picture being qualitatively different from what we actually see.

Comment author: CellBioGuy 16 May 2013 03:05:19AM 0 points [-]

Alternately, the selection could in reality be applied more strongly to helplessness of the infant or shape of the pelvis rather than head size at birth.

Comment author: RolfAndreassen 03 May 2013 04:21:08PM 0 points [-]

I get the impression that deaths from childbirth don't, usually, come about because the head is too large, but because there are other complications: Breech births, especially.

Comment author: ModusPonies 29 April 2013 05:28:00PM 13 points [-]

I am glad to see this report. I've felt that MIRI was producing less cool stuff than I would've expected, but this looks like it will go a long way towards addressing that. I am revising my opinion of the organization upwards. I look forward to reading this, and commit to having done so by the end of this weekend.

Comment author: ciphergoth 04 May 2013 09:14:12AM 3 points [-]

The whole counter-, counter-counter- thing is very difficult to follow. I've seen both you and Dennett use conversations between imagined participants to present such arguments, which I find vastly more readable.

Comment author: ZHD 30 April 2013 06:56:37AM *  3 points [-]

Hominid brain size has not been increasing for at least the past 100,000 years. In fact, the range is tighter and median is lower for homo sapiens vs homo neanderthalensis.

Given that information, how does this change your explanation of your data?

The most important brain developments in the genus have come during the time when brain size was not increasing. This means that size can not be an explanatory variable.

Cheers, ZHD

Comment author: CarlShulman 30 April 2013 08:17:59AM *  3 points [-]

Footnote 44 discusses Neanderthals having larger brains, so it's not new data.

Comment author: ZHD 30 April 2013 08:25:06AM 0 points [-]

Thank you Carl. I am having some difficulty navigating to that discussion. Can you provide a direct link?

Comment author: CarlShulman 30 April 2013 08:27:08AM 1 point [-]

It's the link at the top of the OP. Look on page 38 of the document (page numbers are at the bottom) to find footnote 44.

Comment author: ZHD 30 April 2013 09:00:48AM 5 points [-]

Thanks for the help!

So this is the footnote:

  1. Neanderthals may have had larger brains than modern humans (Ponce de León et al. 2008) and it is an open question how much Neanderthals interbred with the ancestors of modern humans. It is possible that the marginal fitness returns on cognition have leveled off sharply enough that improvements in cognitive efficiency have shifted the total resource cost of brains downward rather than upward over very recent history. If true, this is not the same as Homo sapiens sapiens becoming stupider or even staying the same intelligence. But it does imply that either marginal fitness returns on cognition or marginal cognitive returns on brain scaling have leveled off significantly compared to earlier evolutionary history

That appears to be circular reasoning. It only implies that "marginal fitness return on cognition" has leveled off if we define fitness as a function of brain size—we have no fitness measurement otherwise.

My previous suggestion, that the most important brain developments in our genus are independent of brain size, needs an explanation with a much different anchor.

Comment author: pcm 11 May 2013 07:30:12PM 1 point [-]

There is speculation that brain size decreased due to loss of olfactory and maybe other sensory parts of the brain after dogs took over those functions. See here.

Comment author: Eliezer_Yudkowsky 29 April 2013 12:13:50AM 3 points [-]

Two quick notes on the current text: Kasparov was apparently reading the forum of the opposing players during his chess victory in Kasparov vs. The World, which doesn't quite invalidate the outcome as evidence but does weaken the strength of evidence for cognitive (non-)scaling with population. Also Garett Jones made some relevant remarks here which I should've cited in the discussion of how science scales with scientific population and invested money (or rather, how it doesn't scale).

Comment author: Will_Newsome 30 April 2013 06:24:13PM 2 points [-]

Anyone know the details of Karpov vs. The World? (Here are more GM vs. World games; most involve both sides using stronger-than-human chess engines.)

Comment author: gjm 30 April 2013 11:54:11AM *  6 points [-]

The discussion of Moore's law, faster engineers, hyperbolic growth, etc., seems to me to come close to an important point but not actually engage with it.

As the paper observes, a substantial part of the work of modern CPU design is already done by computers. So why don't we see hyperbolic rather than merely Moorean growth? One reason would be that as long as some fraction of the work, bounded away from zero, is done by human beings, you don't get the superexponential speedup, for obvious Amdahl-ish reasons. The human beings end up being the rate-limiting factor.

Now suppose we take human beings out of the loop entirely. Is the whole thing now in the hands of the ever-speeding-up computers? Alas, no. When some new technology is developed that enables denser circuitry, Intel and their rivals have to actually build the fabs before reaping the benefits by making faster CPUs. And that building activity doesn't speed up exponentially, and indeed its cost increases rapidly from one generation to the next.

There are things that are purely a matter of clever design. For instance, some of the increase in speed of computer programs over the years has come not from the CPUs but from the compilers. But they improve really slowly; hence "Proebsting's Law", coined tongue-in-cheek by Todd Proebsting, that compiler improvements give you a doubling in speed every 18 years. (It's obvious that even if this were meant seriously it couldn't continue at that rate for very long.)

This doesn't refute any claim made in the paper (since it doesn't, e.g., say that actual literal hyperbolic growth is to be expected) but I think it does suggest some downward adjustment in how rapid we should expect self-improvement to be.

Perhaps this sort of thing is discussed later in the paper -- I've only read about the first 1/4. If so, I'll edit or retract this as appropriate when I find it.

[EDITED to add: there is indeed some discussion of this, in section 3.3, around page 48. The arguments there against sensorimotor bottlenecks greatly reducing foominess seem handwavy to me.]

Comment author: jbash 29 April 2013 08:58:22PM 8 points [-]

TL;DR.

The first four or five paragraphs were just bloviation, and I stopped there.

I know you think you can get away with it in "popular education", but if you want to be taken seriously in technical discourse, then you need to rein in the pontification.

Comment author: paper-machine 30 April 2013 03:27:55AM 2 points [-]

if you want to be taken seriously in technical discourse, then you need to rein in the pontification.

I don't disagree, but I also don't think this is correct. There are plenty of verbose mathematicians out there who spend too much time expounding on the philosophical merits of their approach, and they're taken seriously.

You'll excuse me if I don't name names, though.

Comment author: JoshuaZ 30 April 2013 03:33:33AM 9 points [-]

Most of those are people who have already earned it a bit by having major results to their credit.

Comment author: STL 30 April 2013 03:21:50AM 1 point [-]

Can you provide a few examples of sentences that you consider to be particularly empty of meaning?

Comment author: paper-machine 30 April 2013 03:25:40AM 4 points [-]

The first three paragraphs. I think the fourth and fifth paragraphs (contra grandparent) are reasonable, but the biography of I. J. Good doesn't add much support to the thesis.

Comment author: Mitchell_Porter 03 May 2013 07:56:44AM 2 points [-]

Suppose I already believe that, because of computer science, neuroscience, etc, there will in the future be agents or coalitions of agents capable of outwitting human beings for control of the world, and that we can hope to shape the character of these future agents by working on topics like friendly AI. If I already believe that, do I need to read this?

Comment author: Manfred 03 May 2013 08:22:48AM *  1 point [-]

It was fairly interesting, but for me more of an entertainment-budgeted item than a learning-budgeted item.

Comment author: Strilanc 30 April 2013 03:52:40PM *  2 points [-]

I'm going to nitpick on Section 3.8:

If there are several “hard steps” in the evolution of intelligence, then planets on which intelligent life does evolve should expect to see the hard steps spaced about equally across their history, regardless of each step’s relative difficulty. [...]

[...] [...]

[...] the time interval from Australopithecus to Homo sapiens is too short to be a plausible hard step.

I don't think this argument is valid. Assuming there's a last hard step, you'd expect intelligence to show up soon after it was made (because there's no more hard steps).

In terms of the analogy to lock picking, it's inappropriate to set the clock to timeout now. The clock should timeout when it's too late for intelligence to succeed (e.g. when the sun has aged too much and is evaporating the oceans away).

Also, I tested if the claim about the even spacing was correct. The locks do appear to have the same distribution when you condition on being below the time limit. However, the picking times don't appear to be evenly spaced particularly often. With a timeout of 1000 and clocks with pick-chances of 10^-3 through 10^-8 I get average standard deviations between picking times of ~300 which I think implies that the most common situation is for a lock or two to be picked quickly with the other locks consuming all the slack.

edit Now when I read the sentence I see something slightly different. Did you mean "the undertaking of the step should take a long time" or that "the amount of time since the last step was made should be a long time"?

Comment author: ThrustVectoring 29 April 2013 06:52:27PM 2 points [-]

I may have found a minor problem on page 50:

Better algorithms could decrease the serial burden of a thought and allow more thoughts to occur in serial rather than parallel

Shouldn't that be "allow more thoughts to occur in parallel rather than serial"? Turning a thought from multiple parallel sub-tasks to one serial task increases the serial burden of that thought, rather than decreasing it.

Comment author: Eliezer_Yudkowsky 29 April 2013 08:22:44PM 2 points [-]

The idea here is that you use parallelism to implement operations like caching which can decrease the number of serial steps required for a thought, so that more of them can occur one after another. In the simplest case, if you were already using a serial processor to emulate parallel computers, adding parallel power increases serial depth because you need no longer burn serialism to emulate parallelism.

Comment author: ThrustVectoring 29 April 2013 09:07:33PM 0 points [-]

Oh, so diverting serial processing cycles to get serial depth instead of getting half the depth over two independent tasks. I thought the sentence was saying something else entirely: that a better algorithm does the same thing except with higher serial depth over fewer processes.

Comment author: Clippy 29 April 2013 09:24:39PM 2 points [-]

Thanks for adopting my suggestion to publish more on paperclip-production-relevant topics.

Comment author: someonewrongonthenet 30 April 2013 02:24:06AM *  3 points [-]

While I'm amused by your existence, the "novelty account" meme is quite virulent and has the potential to lower the signal-to-noise ratio in the comments if everyone starts doing this...

Comment author: Luke_A_Somers 30 April 2013 05:58:39PM 3 points [-]

There are only so many contextually appropriate novelty names available. It wouldn't make much sense for someone to swoop in and begin posting in character as any random character. So far as I know, we've got Clippy, Voldemort and Quirinus Quirrel, and... one other, maybe?

If people showed up and began noticeably posting in character as random vaguely (anti-)rationality-related characters (Spock, Kamina, Pinkie Pie), we'd have a problem. Fortunately, they don't.

Comment author: army1987 09 June 2013 02:00:16PM 0 points [-]

Death

A few others only wrote one or two comments each.

Comment author: Clippy 30 April 2013 08:42:42PM 3 points [-]

While I'm amused by your account name, the "novelty account" meme is quite virulent and has the potential to lower the signal-to-noise ratio in the comments if everyone starts doing this...

Comment author: paper-machine 30 April 2013 02:48:47AM *  1 point [-]

That's what we said when Clippy first started being ridiculous years ago. Luckily, it constrained itself to a few vanity posts, and the phenomena in general didn't really take off.

Admittedly we're just mostly annoyed that people think we're such an account pretending to be an actual made-of-paper kind of machine, but so it goes.

Comment author: CarlShulman 30 April 2013 04:03:13AM *  1 point [-]

What is the story behind the account name, then?

Comment author: paper-machine 30 April 2013 04:05:11AM 1 point [-]

It's a favorite book of mine.

Comment deleted 30 April 2013 05:35:08PM [-]
Comment deleted 01 May 2013 06:47:25PM [-]
Comment author: leplen 10 June 2013 11:08:37PM *  1 point [-]

I have some questions about the math in the first couple pages, specifically the introduction of k. I'm not totally sure I follow exactly what's going on.

So, my assumption is that we're trying to model AI capacity as a function of investment, and I assume that we're modeling this as the integral of an exponential function of base k such that

with k held constant. The integral is necessary I believe to insure that the derivative of C is positive in both the k<1 and k>1 scenarios. This I believe matches the example of the nuclear chain reaction. I note here that C as I've defined it, is only a function of investment and tells us nothing about time or any other variable. I think it's also true that we've defined C as an exponential because we're assuming that the AI is reinvesting it's returns. This seems to conflict with the linear relationship between investment and returns mentioned in the Chalmer's quote

"The key issue is the “proportionality thesis” saying that among systems of certain class, an increase of δ in intelligence will yield an increase of δ in the intelligence of systems that these systems can design."

although perhaps those deltas are not intended to be quantitative and equal.

But even then, I'm a little uncertain that my relation is correct. It is not clear to me that the sequence of logarithms obtained in the k<1 case is a result of this function. Specifically, I thought the notion of reinvestment was the motivation for choosing an exponential/logarithmic function to start with, and so I'm not clear on why reinvestment suddenly changes the behavior to that of nested logarithms. Is the logarithmic nature of our return being double counted?

I was also confused by the statement

Over the last many decades, world economic growth has been roughly exponential—growth has neither collapsed below exponential nor exploded above, implying a metaphorical k roughly equal to 1

But from my model, which I think is the correct one, this isn't true. I feel like I understand the math from the nuclear chain reaction, but I have

so that k=1 implies not exponential growth, but linear growth. Even worse, no value of k in my model is capable of making k "explode above" the exponential. I agree with the assessment that k has been slightly on the positive side, which gave me some hope I still have the correct model, but then I got really discouraged by the fact that for money is on the order of 1.02 while for the neutrons in the nuclear pile was 1.006. The implication from k values alone is that my bank account is somehow more explosive than a large pile of Uranium. Unfortunately this is not true, and so it seems like my model needs to account not only for C as a function of i, but C as a function of time as well.

This issue really comes into play with the prompt critical AI. One of the ways prompt critical AI is deemed capable of growing exponentially smarter is by stealing access to more hardware. Having this as an option challenges either the definition of investment or seriously challenges the notion of constant k. Even in the limit that solving AI problems is exponentially hard (k<1), if gaining access to more resources/investment is not also exponentially hard, then the AI isn't bounded by how hard it is to solve AI, but by how hard it is to acquire resources. An intelligence fizzle AI that could increase its access to computer time faster than it progressed up the difficulty curve would show exponential growth even IF building an AI was exponentially difficult. Additionally, it's not clear to me that the characterization here only in terms of k, (i.e. return on cognitive investment) and without any regard to time in the original formulation is particularly clear, especially since time dependence seems to be the only difference between scenarios 2 and 3. (Indeed, what is "prompt k" if not k>1 coupled to a short generation time?

I'm really terrible at LW formatting/writing in tiny comment boxes, so if I screwed this up to the point of being confusing let me know.

Comment author: Manfred 01 May 2013 03:42:44AM 1 point [-]

The lock problem from 3.8: Suppose there were 2 locks, one with a uniform solving distribution 5 hours long, and one with a uniform solving distribution 10 hours long. Now suppose we make a new probability distribution where first we solve lock one, then lock two, in times X and Y. The probability is now (up to the time limits) X/10*Y/5. Hey look, symmetry!

Now suppose we condition on the total time being 1 hour. So X+Y=1. But there's still symmetry between X and Y. So yeah.

Comment author: CronoDAS 15 July 2013 04:10:26AM 0 points [-]

When I read this segment, I was compelled to comment:

A key component of the debate between Robin Hanson and myself was the question of locality. Consider: If there are increasing returns on knowledge given constant human brains—this being the main assumption that many non-intelligence-explosion, general technological-hypergrowth models rely on, with said assumption seemingly well-supported by exponential technology-driven productivity growth—then why isn’t the leading human nation vastly ahead of the runner-up economy? Shouldn’t the economy with the most knowledge be rising further and further ahead of its next-leading competitor, as its increasing returns compound?
The obvious answer is that knowledge is not contained within the borders of one country; improvements within one country soon make their way across borders. China is experiencing greater growth per annum than Australia, on the order of 8% versus 3% RGDP growth.92 This is not because technology development in general has diminishing marginal returns. It is because China is experiencing very fast knowledge-driven growth as it catches up to already-produced knowledge that it can cheaply import.

There actually are historical examples of this happening between civilizations that had relatively little information transfer between them. Pre-Columbian America was far behind Europe, as was sub-Saharan Africa. China's economic and technological development has also been relatively isolated from Europe until relatively recently; there were times where it was more advanced and times where it was less advanced.

Comment author: gjm 30 April 2013 10:49:35AM 0 points [-]

Superficial stylistic remark: The paper repeatedly uses the word "agency" where "agent" would seem more appropriate.

Comment author: pcm 11 May 2013 07:10:38PM 0 points [-]

China is experiencing very fast knowledge-driven growth as it catches up to already-produced knowledge that it can cheaply import.

To the extent that AIs other than the most advanced project can generate self-improvements at all, they generate modifications of idiosyncratic code that can’t be cheaply shared with any other AIs.

I say it's at least as expensive for China to import knowledge. A fair amount is trade secrets that are more carefully guarded than AI content. China copies on the order of $1 trillion in value. What's the value of uncopied AI content?

We don’t invest in larger human brains because that’s impossible with current technology

No, we have technology for that (selective breeding, maybe genetic engineering). The return on investment is terrible. In an em dominated world, the technology for building larger minds (and better designed minds) may still be poor compared with the technology for copying. How much will that change with AGI? I expect people to disagree due to differing intuitions about how AGI will work.

Comment author: itaibn0 30 April 2013 10:27:23PM 0 points [-]

One point you don't address: While you justify the claim that intelligence is real thing and can be compared, you don't explain why it would be measurable in a numerical scale. In particular, I don't see what "linear increase in intelligence" and "exponential increase in intelligence" mean and how they can be compared.

Stylistically, I agree with many of the other comments and I think this paper is unsuitable for academic publication. You should keep out discussion of side issues like speculation on the bottlenecks in academic research, how MIRI plans to deal with the potential intelligence explosion, and general discussions on how to reason, and focus just on the arguments for the existence and on the nature of an intelligence explosion.

Comment author: paper-machine 30 April 2013 03:34:25AM *  0 points [-]

There are some places in the text that appear to be originally hyperlinked, but whose hyperlinks are not present in the .pdf. For example, footnote 21.

In general, the paper needs a technical editor.

EDIT: The lack of hyperlinks is clearly something on my end. I apologize for jumping to conclusions.

Comment author: RomeoStevens 30 April 2013 05:46:57AM 2 points [-]

Tangent: Why isn't editing of academic papers done on github or another revision control platform?

Comment author: malo 01 May 2013 08:49:56PM 4 points [-]

MIRI uses Git to track edits for all documents it publishes with it's official template.

Comment author: paper-machine 30 April 2013 06:45:18AM 1 point [-]

Do you mean editing done by the publishers? I don't know anything about that domain.

As far as the writing of academic papers goes, I know a few groups that maintain a CVS, but some portion of mathematicians wouldn't be technical enough to run one. Of the examples I know, two groups only use a CVS because their PI told them to, over much groaning.

Comment author: RomeoStevens 30 April 2013 08:38:31AM *  4 points [-]

I was just struck how this is a perfect example where the efficient flow would go 'see problem->fix problem->submit pull request', rather than having to post here and hoping someone sees it and acts on it.

It occurs to me that the person who makes revision control easy in the same way that social websites made having your own website easy will also make a billion dollars. Dropbox is both a good example of success and a good example of how much more could be done (trying to use dropbox for proper revision control is something I've seen attempted and it is not pretty).

Comment author: David_Gerard 30 April 2013 09:57:01AM *  4 points [-]

It occurs to me that the person who makes revision control easy in the same way that social websites made having your own website easy will also make a billion dollars.

Hell yeah. I have successfully explained to a bunch of not-very-technical managers why they would want version control: "Have you ever spent three days editing the wrong version of a document?" Wide eyes, all slowly nod. Once they understood what it was for, they would have crawled across broken glass for version control. (And since we were using ClearCase, they did pretty much that!)

"Track changes" in Word solves a little of the problem from the other end.

Comment author: paper-machine 30 April 2013 08:46:24AM 2 points [-]

I would anticipate that a github-like approach to editing would get you decent coverage of "local" editing issues (e.g., this hyperlink business) while not obtaining decent coverage of "global" editing issues (e.g., the use of "counter^3-argument", "counter^4-argument" in some places and "counter-counter-counter-argument" in another).

A lot of this just boils down to having an acceptable style guide that an editor can enforce without worrying too much about taking every issue to the author for approval.

Comment author: ZHD 30 April 2013 09:11:21AM 3 points [-]

I like the way you're approaching the problem. However, I think the temptation for a familiar conclusion is too great and that you might be missing some possibilities.

See:

A lot of this just boils down to having an acceptable style guide that an editor can enforce without worrying >too much about taking every issue to the author for approval.

The solution you're putting forth suggests that there needs to be a single person in charge of coalescing the many suggestions and edits.

But the great thing about version control is the ability to branch and tag. There could be an arbitrary number of editors who each have their own branch and set of improvements that they are working on—where non-editor contributors could switch branches and commit changes specific to that branch's needs.

In the end, all branches would need to merge into the trunk. This process doesn't necessarily need a single editor either.

Cheers

Comment author: paper-machine 30 April 2013 02:54:17PM 0 points [-]

One person's "familiar conclusion" is another's "best practices", I suppose.

The solution you're putting forth suggests that there needs to be a single person in charge of coalescing the many suggestions and edits.

Not really. Many suggestions and edits put forth by random people, e.g., here, aren't edits that I think an editor should really make. Nor do I really think a single person is necessary; again, a well-defined style guide would go a long way.

I understand how CVSes work, and I have no problem with collaborative editing. But papers are not coding projects. There are a lot of global things going on that need to happen correctly. Even open source projects tend to have lead developers, no?

Comment author: lukeprog 30 April 2013 04:37:22PM 0 points [-]

The hyperlink in footnote 21 works for me. It goes here. What happens when you click on "this online post" in footnote 21?

We did use a couple editors on the paper, like we do with all our papers.

Comment author: paper-machine 30 April 2013 08:29:15PM *  0 points [-]

What happens when you click on "this online post" in footnote 21?

Nothing. Adobe Reader 11.0.2 on Windows 7.

We did use a couple editors on the paper, like we do with all our papers.

Yeah, I saw the percent signs were interpreted correctly. It's a work in progress.

Comment author: malo 01 May 2013 09:19:16PM *  0 points [-]

MIRI's LaTeX document template uses the /href command to hyperlink text and styles links (both internal and external) using the pdfboarderstyle specification from Abode. We aren't doing anything unusual.

Links are working (and styled) for me in OS X Preview and Adobe Reader 10.1.6, on OS 10.8.3. They even work in Chrome's pdf viewer which currently doesn't support pdfboarderstyle, i.e., the text is linked even though there is no underline or box to indicate that it is.

I suspect something fishy is going on with your Reader install . . .

Also, to clarify Luke's comments, we have a dedicated technical editor (who I have been very impressed with so far), and the papers are reviewed by a couple other people (once they have been typeset) before they are published. I'd be interested to hear about (possibly more appropriate through PM or email) other things in this document that made you think we didn't have a technical editor.

EDIT: I should clarify that the editing and proofreading I'm talking about is done once the content has been finalized. See a definition of technical editing here.

Comment author: paper-machine 01 May 2013 11:36:00PM *  0 points [-]

PM sent.

EDIT: I'm no longer sure that sending all of that over a PM (which I unwisely forgot to retain) was such a great idea. Your edit makes it sound like my objections weren't really under the aegis of "technical editing", but I don't recall objecting to anything that doesn't fall under that objection. Anyone who doubts my sincerity, please feel free to PM me.

Comment author: malo 02 May 2013 06:09:04PM 3 points [-]

Sorry, I didn't mean to imply that at all. I just reread my message an it occurred to me that it might not be clear to everyone what technical editing was.

Your PM was indeed about technical edits.

BTW you can see all PMs you sent by visiting, http://lesswrong.com/message/sent/

Comment author: wedrifid 03 May 2013 03:56:22AM 3 points [-]

BTW you can see all PMs you sent by visiting, http://lesswrong.com/message/sent/

Ohh! Thanks. I hadn't noticed that feature.