Agreeing with several other people that the introduction needs a major rewrite or possibly just a cut. Consider the opening sentence:
Isadore Jacob Gudak, who anglicized his name to Irving John Good and used I. J. Good for publication
Dude, no. Who gives a toss how he anglicised his name? Get to your point, if you have one.
Somewhat similarly, in the fourth paragraph, you have
Please note that...
Please note that the phrase "please note that" is unnecessary; it adds length and the impression that you are snippily correcting someone's blog comment, without adding any information (or politeness) to the sentence. I'm familiar with your argument about formal writing just adding a feeling of authority, but this isn't informality, it's sloppy editing.
Your whole first page, actually, is a pretty good demonstration of not having a point. I get the impression that you thought "Hmm, I need some kind of introduction" and went off to talk about something, anything, that wasn't the actual point of the paper, because the point belongs in the body and not the introduction. This makes for a page that adds nothing. You have a much better introduction starting with the parag...
Superficial stylistic remarks (as you'll see, I've only looked at about the first 1/4 of the paper):
The paper repeatedly uses the word "agency" where "agent" would seem more appropriate.
I agree with paper-machine that the mini-biography of I J Good has little value here.
The remark in section 1 about MIRI being funding-limited is out of place and looks like a whine or a plea for more money. Just take it out.
"albeit" on page 10, shortly before footnote 8, should just be "but". (Or maybe "even though", if that's your meaning.) [EDITED to add: there's another "albeit" that reads oddly to me, in footnote 66 on page 50. It's not wrong, but it feels odd. Roughly, wherever you can correctly write "albeit" you can equivalently write "even though", and that's a funny thing to be starting a footnote with.]
"criteria" in footnote 11 about paperclip maximizers should be "criterion".
In footnote 15 (about "g") the word "entrants" seems very weirdly chosen, and the footnote seems to define g as the observed correlation between different measures of intelligence, which
I agree with paper-machine that the mini-biography of I J Good has little value here.
Done.
The remark in section 1 about MIRI being funding-limited is out of place and looks like a whine or a plea for more money. Just take it out.
Done.
Just a thought on chess playing. Rather than looking at an extreme like Kasparov vs the world, it would be interesting to me to have teams of two, three, and four players of well-known individual ranking. These teams could then play many games against individuals and against each other. The effective ranking of the teams could be determined from their results. In this way, some sense of "how much smarter" a team is than the individual members could be determined. Ideally, the team would not be ranked until it had had significant experience playing as a team. We are interested in what a team could accomplish, and no strong reason to think it would take less time to optimize a team than to optimize an individual.
Along the same lines, teams could be developed to take IQ and other GI correlated tests to see how much smarter a few people together are than a single human. Would the results have implications for optimal AI design?
You correctly describe the problems of coordinating the selection of the best result produced. But there's another big problem: coordinating the division of work.
When you add another player to a huge team of 5000 people, he won't start exploring a completely new series of moves no-one else had considered before. Instead, he will likely spend most of his time considering moves already considered by some of the existing players. That's another reason why his marginal contribution will be so low.
Unlike humans, computers are good at managing divide-and-conquer problems. In chess, a lot of the search for the next move is local in the move tree. That's what makes it a particularly good example of human groups not scaling where computers would.
Having looked through the document again, I feel that a competent technical writer, or anyone with a paper-writing experience, can make this report into a paper suitable for submission within a couple of days, maybe a week, assuming MIRI wants it published. A lot would have to be cut, and the rest rearranged and tidied up, but there is definitely enough meat there for a quality paper or two. I am not sure what MIRI's intention is re this report, other than "hope that the open problems posed therein inspire further work by economists or economically literate modelers".
Page 4: the sum log(w) + log(log(w)) + ... doesn't converge. Some logarithm will be negative and then the next one will be undefined. Presumably you meant to stop the sum once it becomes negative, but then I'm somewhat confused about this argument because I'm not sure it's dimensionally consistent (I'm not sure what units cognitive work is being measured in).
Top of page 18: there's a reference to "this graph" but no graph...?
General comment 1: who's the intended audience here? Most of the paper reads like a blog post, which I imagine could be disconcerting for newcomers trying to evaluate whether they should be paying attention to MIRI and expecting a typical research paper from a fancy-looking .pdf.
General comment 2: I still think this discussion needs more computational complexity. I brought this up to you earlier and I didn't really digest your reply. The question of what you can and can't do with a given amount of computational resources seems highly relevant to understanding what the intelligence explosion could look like; in particular I would be surprised if questions like P vs. NP didn't have a strong bearing on the distribution over timelines (I expect that the faster it is possible to solve NP-complete problems, which apparently includes protein folding, the faster AI could go foom). But then I'm not a domain expert here and I could be off-base for various reasons.
The physical universe doesn't need to "solve" protein folding in the sense of having a worst-case polynomial-time algorithm. It just needs to fold proteins. Many NP-complete problems are "mostly easy" with a few hard instances that rarely come up. (In fact, it's hard to find an NP-complete problem for which random instances are hard: if we could do this, we would use it for cryptography.) It's reasonable to suppose protein folding is like this.
Of course, if this is the case, maybe the AI doesn't care about the rare hard instances of protein folding, either.
can we put an upper bound on the length of proteins using evolutionary time scales?
Not really, most big proteins consist of 'domains' which fold up pretty independantly of each other (smaller proteins can vary quite a bit though). Titin is a ~30,000 amino acid protein in human muscle with ~500 repeats of the same 3 basic modules all laid out in a line... over evolutionary time you can shuffle these functional units around and make all kinds of interesting combinations.
Actually, the lab I'm working in recently had a problem with this. We optimized a gene to be read extremely fast by a ribosome while still producing exactly the same protein sequence (manipulating synonymous codons). But it turned out that when you have the actual protien molecule being extruded from the ribosome as rapidly as we had induced it to be, the normal independant folding of successive domains was disrupted - one domain didn't fully fold before the next domain started being extruded, they interacted, and the protein folded all wrong and didn't work despite having exactly the same sequence as the wild protein.
I think the more important point is that Nature doesn't care about the worst case (if a protein takes forever to fold correctly then it's not going to be of any use). But an AI trying to design arbitrary proteins plausibly might.
In the worst case, the protein folding could be a secure hash
Then it would be harder, in fact impossible, to end up with slightly better proteins via point mutations. A point mutation in a string gives you a completely different secure hash of that string.
This isn't a minor quibble, it's a major reason to go "Whaa?" at the idea that protein folding and protein design have intractable search spaces in practice. They have highly regular search spaces in practice or evolution couldn't traverse that space at all.
This isn't a minor quibble, it's a major reason to go "Whaa?" at the idea that protein folding and protein design have intractable search spaces in practice. They have highly regular search spaces in practice or evolution couldn't traverse that space at all.
Yep, it's highly implausible that a natural non-designed process would happen to be a secure hash (as you know from arguing with cryonics skeptics). And that's before we look at how evolution works.
Good point. The search space is at least smooth once in a few thousand tries. (While doing the nearby fermi estimate, I saw a result that 12% (!!!) of point mutations in some bacteria were beneficial).
That said, the "worst possible case" is usually interesting.
The solution space is large enough that even proteins sampling it's points at a rate of trillions per second couldn't really fold if they were just searching randomly through all possible configurations, that would be NP complete. They don't actually do this of course. Instead they fold piece by piece as they are produced, with local interactions forming domains which tend to retain their approximate structure once they come together to form a whole protein. They don't enter the lowest possible energy state therefore. Prion diseases are an example of what can happen when proteins enter a normally inaccessible local energy minimum, which in that case happens to have a snowballing effect on other proteins.
The result is that they follow grooves in the energy landscape towards an energy well which is robust enough to withstand all sorts of variation, including the horrific inaccuracies of our attempts at modeling. Our energy functions are just very crude approximations to the real one, which is dependent on quantum level effects and therefore intractable. Another issue is that proteins don't fold in isolation - they interact with chaperone proteins and all sorts of other crap. So simu...
Nothing that has physically happened on Earth in real life, such as proteins folding inside a cell, or the evolution of new enzymes, or hominid brains solving problems, or whatever, can have been NP-hard. Period.
I've seen you say this a couple of times, and your interlocutors seem to understand you, even when they dispute your conclusion. But my brain keeps returning an error when I try to parse your claim.
Read literally, "NP-hard" is not a predicate that can be meaningfully applied to individual events. So, in that sense, trivially, nothing that happens (physically or otherwise, if "physically" is doing any work here) can be NP-hard. But you are evidently not making such a trivial claim.
So, what would it look like if the physical universe "solved an NP-hard problem"? Presumably it wouldn't just mean that some actual salesman found a why to use existing airline routes to visit a bunch of pre-specified cities without revisiting any one of them. Presumably it wouldn't just mean that someone built a computer that implements a brute-force exhaustive search for a solution to the traveling salesman problem given an arbitrary graph (a search that the computer will never finish before the heat death of the universe if the example is large). But I can't think of any other interpretation to give to your claim.
Switch the scenario around: if evolution never produced interesting new proteins (anymore, after time T), would that be evidence that there are no other interesting proteins than what evolution produced?
Yes.
That would be evidence that the supply of interesting proteins had been exhausted, just as computer performance at tic-tac-toe and checkers has stopped improving. I don't see where you're coming from here.
Some review notes as I go through it (at a bright dilettante level):
Section 1:
- I wonder if the chain-reaction model is a good one for recursive self-improvement, or is it just the scariest one? What other models have been investigated? For example, the chain-reaction model of financial investment would result in a single entity with the highest return rate dominating the Earth, this has not happened yet, to my knowledge.
Section 1.3:
- There was a recent argument here by David Pearce, I think, that an intelligent enough paperclip maximizer will have to self-modify to be more "humane". If I recall correctly, the logic was that in the process of searching the space of optimization options it will necessarily encounter an imperative against suffering or something to that effect, inevitably resulting in modifying its goal system to be more compassionate, the way humanity seems to be evolving. This would restrict the Orthogonality Thesis to the initial takeoff, and result in goal convergence later on. While this seems like wishful thinking, it might be worth addressing in some detail, beyond the footnote 11.
Chapter 2:
- log(n) + log(log(n)) + ... seems to describe we
For example, the chain-reaction model of financial investment would result in a single entity with the highest return rate dominating the Earth, this has not happened yet, to my knowledge.
Like... humans? Or the way that medieval moneylenders aren't around anymore, and a different type of financial organization seems to have taken over the world instead? See also the discussion of China catching up to Australia.
If I recall correctly, the logic was that in the process of searching the space of optimization options it will necessarily encounter an imperative against suffering or something to that effect, inevitably resulting in modifying its goal system to be more compassionate, the way humanity seems to be evolving.
I see no reason to suspect the space of optimization options contains value imperatives, assuming the AI is guarded against the equivalent of SQL injection attacks.
Humanity seems to be evolving towards compassion because being the causal factors increasing compassion are on average profitable for individual humans with those factors. The easy example of this is stable, strong police forces routinely hanging murderers, instead of those murderers profiting from from their actions. If you don't have an analogue of the police, then you shouldn't expect the analogue of the reduction in murders.
(I should remark that I very much like the way this report is focused; I think that trying to discuss causal models explicitly is much better than trying to make surface-level analogies.)
- empty space for a meditation seems out of place in a more-or-less formal paper
At the very least, using a page break rather than a bunch of ellipses seems better.
The closest thing I have seen to this sort of idea is this:
- log(n) + log(log(n)) + ... seems to describe well the current rate of scientific progress, at least in high-energy physics
I'm going to commit pedantry: nesting enough logarithms eventually gives an undefined term (unless n's complex!). So where Eliezer says "the sequence log(w) + log(log(w)) + log(log(log(w))) will converge very quickly" (p. 4), that seems wrong, although I see what he's getting at.
It really bothers me that he calls it a sequence instead of a series (maybe he means the sequence of partial sums?), and that it's not written correctly.
The series doesn't converge because log(w) doesn't have a fixed point at zero.
It makes sense if you replace log(w) with log^+(w) = max{ log(w), 0 }, which is sometimes written as log(w) in computer science papers where the behavior on (0, 1] is irrelevant.
I suppose that amounts to assuming there's some threshold of cognitive work under which no gains in performance can be made, which seems reasonable.
The discussion of the Yudkowsky-Hanson debate feels rather out of place. The points made are mostly highly relevant to the paper; the fact that they were made during an online debate is less so; the particular language used by either side in that debate still less. This discussion is also particularly informal and blog-post-like (random example: footnote 30), which may or may not be a problem depended on the intended audience for the paper.
I'd recommend major reworking of this section, still addressing the same issues but no longer so concerned with what each party said, or thought, or was happy to concede during that particular debate.
I am glad to see this report. I've felt that MIRI was producing less cool stuff than I would've expected, but this looks like it will go a long way towards addressing that. I am revising my opinion of the organization upwards. I look forward to reading this, and commit to having done so by the end of this weekend.
Comments:
Given human researchers of constant speed, computing speeds double every 18 months.
Human researchers, using top-of-the-line computers as assistants. I get the impression this matters more for chip design than litho-tool design, but it definitely helps for those too.
Humans have around four times the brain volume of chimpanzees, but the difference between us is probably mostly software algorithms.
Is 'software algorithms' the right phrase? I'd characterize the improvements more as firmware or hardware improvements. [edit] Later you use the phrase "cognitive algorithms," which I'm much happier with.
A more concrete example you can use to replace the handwaving: one of the big programming productivity boosters is a second monitor, which seems directly related to low human working memory. It's easy to imagine minds with superior working memory able to handle much more complicated models and tasks. (We indeed seem to see this diversity among humans.)
In particular, your later arguments on serial causal depth seem like they would benefit from explicitly considering working memory as well as speed.
...Any lab that shuts down overnight so its researchers can sleep must
It's easy to imagine minds with superior working memory able to handle much more complicated models and tasks. [..] In particular, your later arguments on serial causal depth seem like they would benefit from explicitly considering working memory
Strong, albeit anecdotal, agreement.
Working memory capacity was a large part of what my stroke damaged, and in colloquial terms I was just stupid, relatively speaking, until that healed/retrained. I was fine when dealing with simple problems, but add even a second level of indirection and I just wasn't able to track. The effect is at least subjectively highly nonlinear.
Incidentally, I think this is the strongest argument against Egan's General Intelligence Theorem (or, alternatively, Deutsch's "Universal Explainer" argument from The Beginning of Infinity). Yes, humans could in theory come up with arbitrarily complex causal models, and that's sufficient to understand an arbitrarily complex causal system, but in practice, unaided humans are limited to rather simple models. Yes, we're very good at making use of aids (I'm reminded of how much writing helps thinking whenever I try to do a complicated calculation in my head), but those limitations represent a plausible way for meaningful superhuman intelligence to be possible.
I hope never to forget the glorious experience of re-inventing the concept of lists, about two weeks into my recovery. I suddenly became indescribably smarter.
In the same vein, I have been patiently awaiting the development of artificial working-memory cognitive buffers. As you say, for practical purposes this is superhuman intelligence.
I am unconvinced by the argument that H. sapiens can't be right at a limit on brain size because some people have larger-than-average heads without their mothers being dead.
Presumably head size is partly determined by environmental factors outside genetic control, and presumably having your mother die in childbirth is a really big disadvantage, much worse than being slightly less intelligent. If that's so, then what should it look like if we, as a species, are hard against that wall? (Which I take to mean that any overall increase in head size would be bad even if being cleverer is a big advantage.) I suggest we'd see head sizes that are far enough away from outright disaster that, even given that random environmental variation, death in childbirth is still pretty rare, but not completely unknown. And, of course, that's just what we see; death in childbirth is very rare now, in prosperous advanced countries, but if you go back 100 years or look in less fortunate parts of the world it's not so rare at all.
This could be quantified, at least kinda. We could look at how the frequency of death in childbirth, in places without modern medical care, varies with head size (though controllin...
A very succinct summary appears in Wikipedia (2013):
Citing Wikipedia in any kind of academic context is generally a bad idea, even if it's just for a summary.
On page 15, you write:
the Moore’s-like law for serial processor speeds broke down in 2004
No citation is given, but I found one: Fuller & Millett (2011). The paper includes this handy graph:
And also this one:
The discussion of Moore's law, faster engineers, hyperbolic growth, etc., seems to me to come close to an important point but not actually engage with it.
As the paper observes, a substantial part of the work of modern CPU design is already done by computers. So why don't we see hyperbolic rather than merely Moorean growth? One reason would be that as long as some fraction of the work, bounded away from zero, is done by human beings, you don't get the superexponential speedup, for obvious Amdahl-ish reasons. The human beings end up being the rate-limiting factor.
Now suppose we take human beings out of the loop entirely. Is the whole thing now in the hands of the ever-speeding-up computers? Alas, no. When some new technology is developed that enables denser circuitry, Intel and their rivals have to actually build the fabs before reaping the benefits by making faster CPUs. And that building activity doesn't speed up exponentially, and indeed its cost increases rapidly from one generation to the next.
There are things that are purely a matter of clever design. For instance, some of the increase in speed of computer programs over the years has come not from the CPUs but from the compiler...
Section 5:
The initial effort to get some numerical models going could be overestimated, unless such models have been done already. At the very least, a small-scale effort can pin-point the hard issues. This reminds me of the core-collapse Supernova modeling: it was reasonably easy to get the explosion modeled, except for the ignition by the initial shock wave. We still don't know what exactly makes them go FOOM. Most models predict a fizzle instead of an explosion. This is likely just a surface analogy, but it might well be that a few months of summer stud...
Two quick notes on the current text: Kasparov was apparently reading the forum of the opposing players during his chess victory in Kasparov vs. The World, which doesn't quite invalidate the outcome as evidence but does weaken the strength of evidence for cognitive (non-)scaling with population. Also Garett Jones made some relevant remarks here which I should've cited in the discussion of how science scales with scientific population and invested money (or rather, how it doesn't scale).
Here is my takeout from the report. It is not a summary, and some of the implications are mine.
The 4 Theses (conjectures, really):
- Inevitability of Intelligence (defined as cross-domain optimization power) Explosion due to recursive self-improvement
- Orthogonality (intelligence/rationality is preference-independent)
- Instrumental Convergence (most optimizers compete for the same resources)
- Complexity of Value (values are not easily formalizable, no 3 laws of robotics)
if true, imply that AGI is an x-risk, because an AGI emerging in an ad hoc fashion wi
I'm also wondering about the estimated FOOM date of 2035 (presumably give or take a decade), is there an explicit calculation of it, and hopefully the confidence intervals as well?
Well, you mentioned on occasion that this date affects your resource allocation between CFAR and MIRI, so it might be a worthwhile exercise to make the calculation explicit and subject to scrutiny, if not in the report, then some place else.
TL;DR.
The first four or five paragraphs were just bloviation, and I stopped there.
I know you think you can get away with it in "popular education", but if you want to be taken seriously in technical discourse, then you need to rein in the pontification.
Most of those are people who have already earned it a bit by having major results to their credit.
Hominid brain size has not been increasing for at least the past 100,000 years. In fact, the range is tighter and median is lower for homo sapiens vs homo neanderthalensis.
Given that information, how does this change your explanation of your data?
The most important brain developments in the genus have come during the time when brain size was not increasing. This means that size can not be an explanatory variable.
Cheers, ZHD
The whole counter-, counter-counter- thing is very difficult to follow. I've seen both you and Dennett use conversations between imagined participants to present such arguments, which I find vastly more readable.
I'm going to nitpick on Section 3.8:
If there are several “hard steps” in the evolution of intelligence, then planets on which intelligent life does evolve should expect to see the hard steps spaced about equally across their history, regardless of each step’s relative difficulty. [...]
[...] [...]
[...] the time interval from Australopithecus to Homo sapiens is too short to be a plausible hard step.
I don't think this argument is valid. Assuming there's a last hard step, you'd expect intelligence to show up soon after it was made (because there's no more ...
I may have found a minor problem on page 50:
Better algorithms could decrease the serial burden of a thought and allow more thoughts to occur in serial rather than parallel
Shouldn't that be "allow more thoughts to occur in parallel rather than serial"? Turning a thought from multiple parallel sub-tasks to one serial task increases the serial burden of that thought, rather than decreasing it.
Suppose I already believe that, because of computer science, neuroscience, etc, there will in the future be agents or coalitions of agents capable of outwitting human beings for control of the world, and that we can hope to shape the character of these future agents by working on topics like friendly AI. If I already believe that, do I need to read this?
I have some questions about the math in the first couple pages, specifically the introduction of k. I'm not totally sure I follow exactly what's going on.
So, my assumption is that we're trying to model AI capacity as a function of investment, and I assume that we're modeling this as the integral of an exponential function of base k such that
=\int{k%5Ei}di=\frac{k%5Ei}{\log(k)})
with k held constant. The integral is necessary I believe to insure that the derivative of C is positive in both the k1 scenarios. This I believe matches the example of the nuclear c...
The lock problem from 3.8: Suppose there were 2 locks, one with a uniform solving distribution 5 hours long, and one with a uniform solving distribution 10 hours long. Now suppose we make a new probability distribution where first we solve lock one, then lock two, in times X and Y. The probability is now (up to the time limits) X/10*Y/5. Hey look, symmetry!
Now suppose we condition on the total time being 1 hour. So X+Y=1. But there's still symmetry between X and Y. So yeah.
...When I read this segment, I was compelled to comment:
A key component of the debate between Robin Hanson and myself was the question of locality. Consider: If there are increasing returns on knowledge given constant human brains—this being the main assumption that many non-intelligence-explosion, general technological-hypergrowth models rely on, with said assumption seemingly well-supported by exponential technology-driven productivity growth—then why isn’t the leading human nation vastly ahead of the runner-up economy? Shouldn’t the economy with the mo
One point you don't address: While you justify the claim that intelligence is real thing and can be compared, you don't explain why it would be measurable in a numerical scale. In particular, I don't see what "linear increase in intelligence" and "exponential increase in intelligence" mean and how they can be compared.
Stylistically, I agree with many of the other comments and I think this paper is unsuitable for academic publication. You should keep out discussion of side issues like speculation on the bottlenecks in academic research, ...
Superficial stylistic remark: The paper repeatedly uses the word "agency" where "agent" would seem more appropriate.
What are the measurement units of "optimization power" and "complex order created per unit time"? What are the typical values for a human?
China is experiencing very fast knowledge-driven growth as it catches up to already-produced knowledge that it can cheaply import.
To the extent that AIs other than the most advanced project can generate self-improvements at all, they generate modifications of idiosyncratic code that can’t be cheaply shared with any other AIs.
I say it's at least as expensive for China to import knowledge. A fair amount is trade secrets that are more carefully guarded than AI content. China copies on the order of $1 trillion in value. What's the value of uncopied AI conte...
There are some places in the text that appear to be originally hyperlinked, but whose hyperlinks are not present in the .pdf. For example, footnote 21.
In general, the paper needs a technical editor.
EDIT: The lack of hyperlinks is clearly something on my end. I apologize for jumping to conclusions.
Summary: Intelligence Explosion Microeconomics (pdf) is 40,000 words taking some initial steps toward tackling the key quantitative issue in the intelligence explosion, "reinvestable returns on cognitive investments": what kind of returns can you get from an investment in cognition, can you reinvest it to make yourself even smarter, and does this process die out or blow up? This can be thought of as the compact and hopefully more coherent successor to the AI Foom Debate of a few years back.
(Sample idea you haven't heard before: The increase in hominid brain size over evolutionary time should be interpreted as evidence about increasing marginal fitness returns on brain size, presumably due to improved brain wiring algorithms; not as direct evidence about an intelligence scaling factor from brain size.)
I hope that the open problems posed therein inspire further work by economists or economically literate modelers, interested specifically in the intelligence explosion qua cognitive intelligence rather than non-cognitive 'technological acceleration'. MIRI has an intended-to-be-small-and-technical mailing list for such discussion. In case it's not clear from context, I (Yudkowsky) am the author of the paper.
Abstract:
The dedicated mailing list will be small and restricted to technical discussants.
This topic was originally intended to be a sequence in Open Problems in Friendly AI, but further work produced something compacted beyond where it could be easily broken up into subposts.
Outline of contents:
1: Introduces the basic questions and the key quantitative issue of sustained reinvestable returns on cognitive investments.
2: Discusses the basic language for talking about the intelligence explosion, and argues that we should pursue this project by looking for underlying microfoundations, not by pursuing analogies to allegedly similar historical events.
3: Goes into detail on what I see as the main arguments for a fast intelligence explosion, constituting the bulk of the paper with the following subsections:
4: A tentative methodology for formalizing theories of the intelligence explosion - a project of formalizing possible microfoundations and explicitly stating their alleged relation to historical experience, such that some possibilities can allegedly be falsified.
5: Which open sub-questions seem both high-value and possibly answerable.
6: Formally poses the Open Problem and mentions what it would take for MIRI itself to directly fund further work in this field.