I've put far more time that most engaging your singularity arguments, my responses have consisted of a lot more than just projecting a new growth jump from stats on the last three jumps, and I've continue to engage the topic long after that June 2008 post. So it seems to me unfair to describe me as someone arguing "for ending argument-gathering" on the basis that this one projection says all that can be said.
I agree that inside vs outside viewing is a continuum, not a dichotomy. I'd describe the key parameter as the sort of abstractions used, and the key issue is how well grounded are those abstractions. Outside views tend to pretty directly use abstractions that are based more "surface" features of widely known value. Inside views tend to use more "internal" abstractions, and inferences with longer chains.
Responding most directly to your arguments my main critiques have been about the appropriateness of your abstractions. You may disagree with them, and think I try to take the conversation in the wrong direction, but I don't see how I can be described as trying to halt the conversation.
I don't see how I can be described as trying to halt the conversation.
Allow me to disclaim that you usually don't. But that particular referenced post did, and taw tried it even more blatantly - to label further conversation as suspect, ill-advised, and evidence of morally nonvirtuous (giving in to pride and the temptation to show off) "inside viewing". I was frustrated with this at the time, but had too many higher-priority things to say before I could get around to describing exactly what frustrated me about it.
It's also not clear to me how you think someone should be allowed to proceed from the point where you say "My abstractions are closer to the surface than yours, so my reference class is better", or if you think you just win outright at that point. I tend to think that it's still a pretty good idea to list out the underlying events being used as alleged evidence, stripped of labels and presented as naked facts, and see how much they seem to tell us about the future event at hand, once the covering labels are gone. I think that under these circumstances the force of implication from agriculture to self-improving AI tends to sound pretty weak.
I've got no problem with it phrased that way. To be clear, the part that struck me as unfair was this:
Excess inside viewing usually continues even after folks are warned that outside viewing works better; after all, inside viewing better show offs inside knowledge and abilities. People usually justify this via reasons why the current case is exceptional. (Remember how all the old rules didn’t apply to the new dotcom economy?) So expect to hear excuses why the next singularity is also an exception where outside view estimates are misleading. Let’s keep an open mind, but a wary open mind.
Why isn't the obvious outside view to draw a line showing the increased peacefulness of contacts with the increasing technological development of the parties involved, and extrapolate to super-peaceful aliens? Isn't this more or less exactly why you argue that AIs will inevitably trade with us? Why extrapolate for AIs but not for aliens?
To be clear on this, I don't simply distrust the advice of an obvious outside view, I think that in cases like these, people perform a selective search for a reference class that supports a foregone conclusion (and then cry "Outside View!"). This foregone conclusion is based on inside viewing in the best case; in the worst case it is based entirely on motivated cognition or wishful thinking. Thus, to cry "Outside View!" is just to conceal the potentially very flawed thinking that went into the choice of reference class.
Forbidding "reliance" on pain of loss of prestige isn't all that much better than forbidding "exploration" on pain of loss of prestige. People are allowed to talk about my arguments but of course not take them seriously? Whereas it's perfectly okay to rely on your "outside view" estimate? I don't think the quoted paragraph is one I can let stand no matter how you reframe it...
"inside view" and "outside view" seem misleading labels for things that are actually "bayesian reasoning" and "bayesian reasoning deliberately ignoring some evidence to account for flawed cognitive machinery". The only reason for applying the "outside view" is to compensate for our flawed machinery, so to attack an "inside view", one needs to actually give a reasonable argument that the inside view has fallen prey to bias. This argument should come first, it should not be assumed.
We say: "Would you care to make a side bet on that?"
And I'd say . . . . "Sure! I recognize that I normally plan to finish 9 to 10 days early to ensure that I finish before the deadline and that I normally "fail" and only finish a day or two early (but still succeed at the real deadline) . . . . but now, you've changed the incentive structure (i.e. the entire problem) so I will now plan to finish 9 or 10 days before my new deadline (necessary to take your money) of 9 or 10 days before the real deadline. Are you sure that you really want to make that side bet?
I note also that "Would you care to make a side bet on that? is interesting as a potential conversation-filter but can also, unfortunately, act as a conversation-diverter.
Eliezer, the 'outside view' concept can also naturally be used to describe the work of Philip Tetlock, who found that political/foreign affairs experts were generally beaten by what Robin Dawes calls the "the robust beauty of simple linear models." Experts relying on coherent ideologies (EDIT: hedgehogs) did particularly badly.
Those political events were affected by big systemic pressures that someone could have predicted using inside view considerations, e.g. understanding the instability of the Soviet Union, but in practice acknowledged experts were not good enough at making use of such insights to generate net improvements on average.
Now, we still need to assign probabilities over different different models, not all of which should be so simple, but I think it's something of a caricature to focus so much on the homework/curriculum planning problems.
(It's foxes who know many things and do better; the hedgehog knows one big thing.)
I haven't read Tetlock's book yet. I'm certainly not surprised to hear that foreign affairs "experts" are full of crap on average; their incentives are dreadful. I'm much more surprised to hear that situations like the instability of the Soviet Union could be described and successfully predicted by simple linear models, and I'm extremely suspicious if the linear models were constructed in retrospect. Wasn't this more like the kind of model-based forecasting that was actually done in advance?
Conversely if the result is just that hedgehogs did worse than foxes, I'm not surprised because hedgehogs have worse incentives - internal incentives, that is, there are no external incentives AFAICT.
I have read Dawes on medical experts being beaten by improper linear models (i.e., linear models with made-up -1-or-1 weights and normalized inputs, if I understand correctly) whose factors are the judgments of the same experts on the facets of the problem. This ought to count as the triumph or failure of something but it's not quite isomorphic to outside view versus inside view.
I think there probably is a good reference class for predictions surrounding the singularity. When you posted on "what is wrong with our thoughts? you identified it: the class of instances of the human mind attempting to think and act outside of its epistemologically nurturing environment of clear feedback from everyday activities.
See, e.g. how smart humans like Stephen Hawking, Ray Kurzweil, Kevin Warwick, Kevin Kelly, Eric Horowitz, etc have all managed to say patently absurd things about the issue, and hold mutually contradictory positions, with massive overconfidence in some cases. I do not exclude myself from the group of people who have said absurd things about the Singularity, and I think we shouldn't exclude Eliezer either. At least Eliezer has put in massive amounts of work for what may well be the greater good of humanity, which is morally commendable.
To escape from this reference class, and therefore from the default prediction of insanity, I think that bringing in better feedback and a large diverse community of researchers might work. Of course, more feedback and more researchers = more risk according to our understanding of AI motivations. But ultimately, that's an unavoidable trade-off; the lone madman versus the global tragedy of the commons.
Large communities don't constitute help or progress on the "beyond the realm of feedback" problem. In the absence of feedback, how is a community supposed to know when one of its members has made progress? Even with feedback we have cases like psychotherapy and dietary science where experimental results are simply ignored. Look at the case of physics and many-worlds. What has "diversity" done for the Singularity so far? Kurzweil has gotten more people talking about "the Singularity" - and lo, the average wit of the majority hath fallen. If anything, trying to throw a large community at the problem just guarantees that you get the average result of failure, rather than being able to notice one of the rare individuals or minority communities that can make progress using lower amounts of evidence.
I may even go so far as to call "applause light" or "unrelated charge of positive affect" on the invocation of a "diverse community" here, because of the degree to which the solution fails to address the problem.
Hanson's argument was interesting but ultimately I think it's just numerology - there's no real physical reason to expect that pattern to continue, especially given how different/loosely-related the 3 previous changes were.
This is an excellent post, thank you.
An earlier comment of yours pointed out that one compensates for overconfidence not by adjusting ones probability towards 50%, but by adjusting it towards the probability that a broader reference class would give. In this instance, the game of reference class tennis seems harder to avoid.
It seems like the Outside View should only be considered in situations which have repeatably provided consistent results. This is an example of the procrastinating student. The event has been repeated numerous time with closely similar outcomes.
If the data is so insufficient that you have a hard time casting it to a reference class, that would imply that you don't have enough examples to make a reference and that you should find some other line of argument.
This whole idea of outside view is analogous to instance based learning or case based reasoning. Y...
I may be overlooking something, but I'd certainly consider Robin's estimate of 1-2 week doublings a FOOM. Is that really a big difference compared with Eliezer's estimates? Maybe the point in contention is not the time it takes for super-intelligence to surpass human ability, but the local vs. global nature of the singularity event; the local event taking place in some lab, and the global event taking place in a distributed fashion among different corporations, hobbyists, and/or governments through market mediated participation. Even this difference isn...
If the Japanese students had put as much effort into their predictions as Eliezer has put into thinking about the singularity then I dare say they would have been rather more accurate, perhaps even more so than the "outside view" prediction.
I prefer the outside view when speaking with good friends, because they know me well enough to gather what I'm really saying isn't 'Stop Here!' but rather 'Explain to me why I shouldn't stop here?'
Perhaps this isn't really the outside view but the trappings of the outside view used rhetorically to test whether the other party is willing to put some effort into explaining their views. The Outside View as a test of your discussion partner.
The Inside View can be a conversation halter as well; going 'farther inside' or 'farther outside' than your partner can d...
Someone should point out that "That is a conversation-halter" is very often a conversation halter.
The outside view technique is as follows:
You are given an estimation problem f(x)=?. x is noisy and you don't know all of the internals of f. First choose any set of functions F containing f. Then find a huge subset G of F such that g in G has that for all y in Y, g(y) is (say) bounded to some nice range R. Now find your probability p that x is in Y and your probability q that f is in G. Then with probability p*q f(x) is in R and this particular technique says nothing about f(x) in the remaining 1-p*q of your distribution.
Sometimes this is extremely helpfu...
Again, even if I don't have time for it myself, I think it would be useful to gather data on particular disagreements based on such "outside views" with vague similarities, and then see how often each side happened to be right.
Of course, even if the "outside view" happened to be right in most such cases, you might just respond with the same argument that all of these particular disagreements are still radically different from your case. But it might not be a very good response, in that situation.
I always love reading Less Wrong. I am just sometimes confused, for many days, about what exactly I have read. Until, something pertinent comes along and reveals the salience of what I had read, and then I say "OH! Now I get it!"
At present, I am between those two states... Waiting for the Now I get it moment.
"Eventually, some AI will go FOOM, locally self-improvingly rather than global-economically"
Ouch. This statement smells to me like backtracking from your actual position. If you honestly have no time estimate beyond "eventually", why does the SIAI exist? Don't you have any more urgent good things to do?
(edited to remove unrelated arguments, there will be time for them later)
I honestly don't understand your objection. "Eventually" means "sometime between tomorrow and X years from now" where my probability distribution over X peaks in the 20-40 range and then starts dropping off but with a long tail because hey, gotta widen those confidence intervals.
If I knew for an absolute fact that nothing was going to happen for the next 100 years, it would still be a pretty damned urgent problem, you wouldn't want to just let things slide until we ended up in a position as awful as the one we probably occupy in real life.
I still feel shocked when I read something like this and remember how short people's time horizons are, how they live in a world that is so much tinier than known space and time, a world without a history or a future or an intergalactic civilization that bottlenecks through it. Human civilization has been around for thousands of years. Anything within the next century constitutes the last minutes of the endgame.
It's a bit off topic, but I've been meaning to ask Eliezer this for a while. I think I get the basic logic behind "FOOM." If a brain as smart as ours could evolve from pretty much nothing, then it seems likely that sooner or later (and I have not the slightest idea whether it will be sooner or later) we should be able to use the smarts we have to design a mind that is smarter. And if we can make a mind smarter than ours, it seems likely that that mind should be able to make one smarter than it, and so on. And this process should be pretty explosi...
Once again, Bayesian reasoning comes to the rescue. The assertion to stop updating based on new data (ignore the inside view!) is just plain wrong.
However a reminder to be careful and objective about the probability one might assign to a new bit of data (Inside view data is not privileged over outside view data! And it might be really bad!) is helpful.
The assertion to stop updating based on new data (ignore the inside view!) is just plain wrong.
I'd like to be able to say that, but there actually is research showing how human beings get more optimistic about their Christmas shopping estimates as they try to visualize the details of when, where, and how.
Your statement is certainly true of an ideal rational agent, but it may not be carried in human practice.
I see a couple of problems with classifying "intelligent design by machines" as a big evolutionary leap away from "intelligent design by humans".
The main difference is one of performance. Performance has been increasing gradually anyway - what is happening now is that it is now increasing faster.
Also, humans routinely augment their intelligence by using machine tools - blurring any proposed line between the machine-augmented humans that we have today and machine intelligence.
My favoured evolutionary transition classification scheme is ...
You seem to be phrasing this as an either/or decision.
Remembering that all decisions are best formalized as functions, the outside view asks, "What other phenomena are like this one, and what did their functions look like?" without looking at the equation. The inside view tries to analyze the function. The hardcore inside view tries to actually plot points on the function; the not-quite-so-ambitious inside view just tries to find its zeros, inflection points, regions where it must be positive or negative, etc.
For complex issues, you should go b...
The "Outside View" is the best predictor of timeframes and happiness (Dan Gilbert 2007)
The reason why people have choosen to study students and what they think about homework is probably because they supposed there were mistakes in average students declarations. Same for studying affective forecast (technical term for 'predicting future happiness given X').
I suspect this kind of mistake does not happen in engineering companies when they evaluate next year's profits given their new technical environment... Therefore no one bothers to study engineers inside views.
I see a couple of problems with classifying "intelligent design by machines" as a big leap away from "intelligent design by humans".
The main difference is one of performance. Performance has been increasing gradually anyway - what is happening now is that it is now increasing faster.
Also, humans routinely augment their intelligence by using machine tools - blurring the proposed line between human and machine intelligence.
My favoured evolutionary transition classification scheme is to bundle the modern changes together - and describe th...
Followup to: The Outside View's Domain, Conversation Halters
Reply to: Reference class of the unclassreferenceable
In "conversation halters", I pointed out a number of arguments which are particularly pernicious, not just because of their inherent flaws, but because they attempt to chop off further debate - an "argument stops here!" traffic sign, with some implicit penalty (at least in the mind of the speaker) for trying to continue further.
This is not the right traffic signal to send, unless the state of knowledge is such as to make an actual halt a good idea. Maybe if you've got a replicable, replicated series of experiments that squarely target the issue and settle it with strong significance and large effect sizes (or great power and null effects), you could say, "Now we know." Or if the other is blatantly privileging the hypothesis - starting with something improbable, and offering no positive evidence to believe it - then it may be time to throw up hands and walk away. (Privileging the hypothesis is the state people tend to be driven to, when they start with a bad idea and then witness the defeat of all the positive arguments they thought they had.) Or you could simply run out of time, but then you just say, "I'm out of time", not "here the gathering of arguments should end."
But there's also another justification for ending argument-gathering that has recently seen some advocacy on Less Wrong.
An experimental group of subjects were asked to describe highly specific plans for their Christmas shopping: Where, when, and how. On average, this group expected to finish shopping more than a week before Christmas. Another group was simply asked when they expected to finish their Christmas shopping, with an average response of 4 days. Both groups finished an average of 3 days before Christmas. Similarly, Japanese students who expected to finish their essays 10 days before deadline, actually finished 1 day before deadline; and when asked when they had previously completed similar tasks, replied, "1 day before deadline." (See this post.)
Those and similar experiments seem to show us a class of cases where you can do better by asking a certain specific question and then halting: Namely, the students could have produced better estimates by asking themselves "When did I finish last time?" and then ceasing to consider further arguments, without trying to take into account the specifics of where, when, and how they expected to do better than last time.
From this we learn, allegedly, that "the 'outside view' is better than the 'inside view'"; from which it follows that when you're faced with a difficult problem, you should find a reference class of similar cases, use that as your estimate, and deliberately not take into account any arguments about specifics. But this generalization, I fear, is somewhat more questionable...
For example, taw alleged upon this very blog that belief in the 'Singularity' (a term I usually take to refer to the intelligence explosion) ought to be dismissed out of hand, because it is part of the reference class "beliefs in coming of a new world, be it good or evil", with a historical success rate of (allegedly) 0%.
Of course Robin Hanson has a different idea of what constitutes the reference class and so makes a rather different prediction - a problem I refer to as "reference class tennis":
If I were to play the game of reference class tennis, I'd put recursively self-improving AI in the reference class "huge mother#$%@ing changes in the nature of the optimization game" whose other two instances are the divide between life and nonlife and the divide between human design and evolutionary design; and I'd draw the lesson "If you try to predict that things will just go on sorta the way they did before, you are going to end up looking pathetically overconservative".
And if we do have a local hard takeoff, as I predict, then there will be nothing to say afterward except "This was similar to the origin of life and dissimilar to the invention of agriculture". And if there is a nonlocal economic acceleration, as Robin Hanson predicts, we just say "This was similar to the invention of agriculture and dissimilar to the origin of life". And if nothing happens, as taw seems to predict, then we must say "The whole foofaraw was similar to the apocalypse of Daniel, and dissimilar to the origin of life or the invention of agriculture". This is why I don't like reference class tennis.
But mostly I would simply decline to reason by analogy, preferring to drop back into causal reasoning in order to make weak, vague predictions. In the end, the dawn of recursive self-improvement is not the dawn of life and it is not the dawn of human intelligence, it is the dawn of recursive self-improvement. And it's not the invention of agriculture either, and I am not the prophet Daniel. Point out a "similarity" with this many differences, and reality is liable to respond "So what?"
I sometimes say that the fundamental question of rationality is "Why do you believe what you believe?" or "What do you think you know and how do you think you know it?"
And when you're asking a question like that, one of the most useful tools is zooming in on the map by replacing summary-phrases with the concepts and chains of inferences that they stand for.
Consider what inference we're actually carrying out, when we cry "Outside view!" on a case of a student turning in homework. How do we think we know what we believe?
Our information looks something like this:
Therefore, when new student X279 comes along, even though we've never actually tested them before, we ask:
"How long before deadline did you plan to complete your last three assignments?"
They say: "10 days, 9 days, and 10 days."
We ask: "How long before did you actually complete them?"
They reply: "1 day, 1 day, and 2 days".
We ask: "How long before deadline do you plan to complete this assignment?"
They say: "8 days."
Having gathered this information, we now think we know enough to make this prediction:
"You'll probably finish 1 day before deadline."
They say: "No, this time will be different because -"
We say: "Would you care to make a side bet on that?"
We now believe that previous cases have given us strong, veridical information about how this student functions - how long before deadline they tend to complete assignments - and about the unreliability of the student's planning attempts, as well. The chain of "What do you think you know and how do you think you know it?" is clear and strong, both with respect to the prediction, and with respect to ceasing to gather information. We have historical cases aplenty, and they are all as similar to each other as they are similar to this new case. We might not know all the details of how the inner forces work, but we suspect that it's pretty much the same inner forces inside the black box each time, or the same rough group of inner forces, varying no more in this new case than has been observed on the previous cases that are as similar to each other as they are to this new case, selected by no different a criterion than we used to select this new case. And so we think it'll be the same outcome all over again.
You're just drawing another ball, at random, from the same barrel that produced a lot of similar balls in previous random draws, and those previous balls told you a lot about the barrel. Even if your estimate is a probability distribution rather than a point mass, it's a solid, stable probability distribution based on plenty of samples from a process that is, if not independent and identically distributed, still pretty much blind draws from the same big barrel.
You've got strong information, and it's not that strange to think of stopping and making a prediction.
But now consider the analogous chain of inferences, the what do you think you know and how do you think you know it, of trying to take an outside view on self-improving AI.
What is our data? Well, according to Robin Hanson:
From this, Robin extrapolates, the next big growth mode will have a doubling time of 1-2 weeks.
So far we have an interesting argument, though I wouldn't really buy it myself, because the distances of difference are too large... but in any case, Robin then goes on to say: We should accept this estimate flat, we have probably just gathered all the evidence we should use. Taking into account other arguments... well, there's something to be said for considering them, keeping an open mind and all that; but if, foolishly, we actually accept those arguments, our estimates will probably get worse. We might be tempted to try and adjust the estimate Robin has given us, but we should resist that temptation, since it comes from a desire to show off insider knowledge and abilities.
And how do we know that? How do we know this much more interesting proposition that it is now time to stop and make an estimate - that Robin's facts were the relevant arguments, and that other arguments, especially attempts to think about the interior of an AI undergoing recursive self-improvement, are not relevant?
Well... because...
It seems to me that once you subtract out the scary labels "inside view" and "outside view" and look at what is actually being inferred from what - ask "What do you think you know and how do you think you know it?" - that it doesn't really follow very well. The Outside View that experiment has shown us works better than the Inside View, is pretty far removed from the "Outside View!" that taw cites in support of predicting against any epoch. My own similarity metric puts the latter closer to the analogies of Greek philosophers, actually. And I'd also say that trying to use causal reasoning to produce weak, vague, qualitative predictions like "Eventually, some AI will go FOOM, locally self-improvingly rather than global-economically" is a bit different from "I will complete this homework assignment 10 days before deadline". (The Weak Inside View.)
I don't think that "Outside View! Stop here!" is a good cognitive traffic signal to use so far beyond the realm of homework - or other cases of many draws from the same barrel, no more dissimilar to the next case than to each other, and with similarly structured forces at work in each case.
After all, the wider reference class of cases of telling people to stop gathering arguments, is one of which we should all be wary...