lukeprog comments on Do Earths with slower economic growth have a better chance at FAI? - Less Wrong

30 Post author: Eliezer_Yudkowsky 12 June 2013 07:54PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (174)

You are viewing a single comment's thread. Show more comments above.

Comment author: Eliezer_Yudkowsky 14 June 2013 02:40:10PM 11 points [-]

General remark: At some point I need to write a post about how I'm worried that there's an "unpacking fallacy" or "conjunction fallacy fallacy" practiced by people who have heard about the conjunction fallacy but don't realize how easy it is to take any event, including events which have already happened, and make it look very improbable by turning one pathway to it into a large series of conjunctions. E.g. I could produce a long list of things which allegedly have to happen for a moon landing to occur, some of which turned out to not be necessary but would look plausible if added to the list ante facto, with no disjunctive paths to the same destination, and thereby make it look impossible. Generally this manifests when somebody writes a list of alleged conjunctive necessities, and I look over the list and some of the items seem unnecessary (my model doesn't go through them at all), obvious disjunctive paths have been omitted, the person has assigned sub-50% probability to things that I see as mainline 90% probabilities, and conditional probabilities when you assume the theory was right about 1-N would be significantly higher for N+1. Most of all, if you imagine taking the negation of the assertion and unpacking it into a long list of conjunctive probabilities, it would look worse - there should be a name for the problem of showing that X has weak arguments but not considering that ~X has even weaker arguments. Or on a meta level, since it is very easy to make things look more conjunctive, we should perhaps not be prejudiced against things which somebody has helpfully unpacked for us into a big conjunction, when the core argument still seems pretty simple on some level.

When I look over this list, my reaction is that:

(1) is a mainline assumption with odds of 5:1 or 10:1 - of course future intergalactic civilization bottlenecks through the goals of a self-improving agency, how would you get to an intergalactic civilization without that happening? If this accounts for much of our disagreement then we're thinking about entirely different scenarios, and I'm not sure how to update from your beliefs about mostly scenario B to my beliefs about mostly scenario A. It makes more sense to call (1) into question if we're really asking about global vs. local, but then we get into the issue of whether global scenarios are mostly automatic losses anyway. If (1) is really about whether we should be taking into account a big chunk of survivable global scenarios then this goes back to a previous persistent disagreement.

(2) I don't see the relevance - why does a long time horizon vs. a short time horizon matter? 80 years would not make me relax and say that we had enough serial depth, though it would certainly be good news ceteris paribus, there's no obvious threshold to cross.

Listing (3) and (4) as separate items was what originally made my brain shout "unpacking fallacy!" There are several subproblems involved in FAI vs. UFAI, of which the two obvious top items are the entire system being conducive to goal stability through self-improvement which may require deducing global properties to which all subsystems must be conducive, and the goal loading problem. These both seem insight-heavy which will require serial time to solve. The key hypothesis is just that there are insight-heavy problems in FAI which don't parallelize well relative to the wide space of cobbled-together designs which might succeed for UFAI. Odds here are less extreme than for (1) but still in the range of 2:1-4:1. The combined 3-4 issue is the main weak point, but the case for "FAI would parallelize better than UFAI" is even weaker.

(5) makes no sense to ask as a conditionally independent question separate from (1); if (1) is true then the only astronomical effects of modern-day economic growth are whatever effects that growth has on AI work, and to determine if economic growth is qualitatively good or bad, we ask about the sign of the effect neglecting its magnitude. I suppose if the effect were trivial enough then we could just increase the planet's growth rate by 5% for sheer fun and giggles and it would have no effect on AI work, but this seems very unlikely; a wealthier planet will ceteris paribus have more AI researchers. Odds of 10:1 or better.

On net, this says that in my visualization the big question is just "Does UFAI parallelize better than FAI, or does FAI parallelize better than UFAI?" and we find that the case for the second clause is weaker than the first; or equivalently "Does UFAI inherently require serial time more than FAI requires serial time?" is weaker than "Does FAI inherently require serial time more than UFAI requires serial time?" This seems like a reasonable epistemic state to me.

The resulting shove at the balance of the sign of the effect of economic growth would have to be counterbalanced by some sort of stronger shove in the direction of modern-day economic growth having astronomical benefits. And the case for e.g. "More econ growth means friendlier international relations and so they endorse ideal Y which leads them to agree with me on policy Z" seems even more implausible when unpacked into a series of conjunctions. Lots of wealthy people and relatively friendly nations right now are not endorsing policy Z.

To summarize and simplify the whole idea, the notion is:

Right now my estimate of the sign of the astronomical effect of modern-day economic growth is dominated by a 2-node conjunction of, "Modern-day econ growth has a positive effect on resources into both FAI and UFAI" and "The case for FAI parallelizing better than UFAI is weaker than the converse case". For this to be not true requires mainly that somebody else demonstrate an effect or set of effects in the opposite direction which has better net properties after its own conjunctions are taken into account. The main weakness in the argument and lingering hope that econ growth is good, isn't that the original argument is very conjunctive, but rather it's that faster econ growth seems like it should have a bunch of nice effects on nice things and so the disjunction of other econ effects might conceivably swing the sign the other way. But it would be nice to have at least one plausible such good effect without dropping our standards so low that we could as easily list a dozen equally (im)plausible bad effects.

Even without doing any calculations, it is extraordinarily hard to imagine that the difference between "world at war" and "world at peace" is less than the difference between "world with slightly more parallelization in AI work" and "world with slightly less parallelization;"

With small enough values of 'slightly' obviously the former will have a greater effect, the question is the sign of that effect; also it's not obvious to me that moderately lower amounts of econ growth lead to world wars, and war seems qualitatively different in many respects from poverty. I also have to ask if you are possibly maybe being distracted by the travails of one planet as a terminal value, rather than considering that planet's instrumental role in future galaxies.

Comment author: lukeprog 04 October 2013 02:14:56AM *  4 points [-]

At some point I need to write a post about how I'm worried that there's an "unpacking fallacy" or "conjunction fallacy fallacy" practiced by people who have heard about the conjunction fallacy but don't realize how easy it is to take any event, including events which have already happened, and make it look very improbable by turning one pathway to it into a large series of conjunctions.

Related: There's a small literature on what Tversky called "support theory," which discusses packing and unpacking effects: Tversky & Koehler (1994); Ayton (1997); Rottenstreich & Tversky (1997); Macchi et al. (1997); Fox & Tversky (1998); Brenner & Koehler (1999); Chen et al. (2001); Boven & Epley (2003); Brenner et al. (2005); Bligin & Brenner (2008).

Comment author: ChrisHallquist 18 October 2013 08:33:47AM 4 points [-]

Luke asked me to look into this literature for a few hours. Here's what I found.

The original paper (Tversky and Koehler 1994) is about disjunctions, and how unpacking them raises people’s estimate of the probaility. So for example, asking people to estimate the probability someone died of “heart disease, cancer, or other natural causes” yields a higher probability estimate than if you just ask about “natural causes.”

They consider the hypothesis this might be because they take the researcher’s apparent emphasis as evidence that’s it’s more likely, but they tested & disconfirmed this hypothesis by telling people to take the last digit of their phone number and estimate the percentage of couples that have that many children. Percentages sum to greater than 1.

Finally, they check whether experts are vulnerable to this bias by doing an experiment similar to the first experiment, but using physicians at Stanford University as the subjects and asking them about a hypothetical case of a woman admitted to an emergency room. They confirmed that yes, experts are vulnerable to this mistake too.

This phenomenon is known as “subadditivity.” A subsequent study (RottenStreich and Tversky 1997) found that subadditivity can even occur when dealing with explicit conjunctions. Macci et al. (1999) found evidence of superadditivity: ask some people how probable it is that the freezing point of alcohol is below that of gasoline, other people how probable it is that the freezing point of gasoline is below that of alcohol, average answers sum to less than 1.

Other studies try to refine the mathematical model of how people make judgements in these kinds of cases, but the experiments I’ve described are the most striking empirical results, I think. One experiment that talks about unpacking conjunctions (rather than disjunctions, like the experiments I’ve described so far) is Boven and Epley (2003, particularly their first experiment, where they ask people how much an oil refinery should be punished for pollution. This pollution is described either as leading to an increase in “asthma, lung cancer, throat cancer, or all varieties of respiratory diseases,” or just as leading to an increase in “all varieties of respiratory diseases.” In the first condition, people want to punish refinery more. But, in spite of being notably different from previous unpacking experiments, still not what Eliezer was talking about.

Below are some other messy notes I took:

http://commonsenseatheism.com/wp-content/uploads/2013/10/Fox-Tversky-A-belief-based-account-of-decision-under-uncertainty.pdf Uses support theory to develop account of decision under uncertainty.

http://commonsenseatheism.com/wp-content/uploads/2013/10/Brenner-Koehler-Subjective-probability-of-disjunctive-hypotheses-local-weight-models-for-decomposition-and-evidential-support.pdf Something about local weights; didn't look at this one much.

http://commonsenseatheism.com/wp-content/uploads/2013/10/Chen-et-al-The-relation-between-probability-and-evidence-judgment-an-extension-of-support-theory.pdf Tweaking math behind support theory to allow for superadditivity.

http://commonsenseatheism.com/wp-content/uploads/2013/10/Brenner-et-al-Modeling-patterns-of-probability-calibration-with-random-support-theory.pdf Introduces notion of random support theory.

http://bear.warrington.ufl.edu/brenner/papers/bilgin-brenner-jesp08.pdf Unpacking effects weaker when dealing with near future as opposed to far future.

Other articles debating how to explain basic support theory results: http://bcs.siu.edu/facultypages/young/JDMStuff/Sloman%20(2004)%20unpacking.pdf http://aris.ss.uci.edu/~lnarens/Submitted/problattice11.pdf http://eclectic.ss.uci.edu/~drwhite/pw/NarensNewfound.pdf

Comment author: Nick_Beckstead 29 October 2014 04:41:58PM 0 points [-]

What this shows is that people are inconsistent in a certain way. If you ask them the same question in two different ways (packed vs. unpacked) you get different answers. Is there any indication of which is the better way to ask the question, or whether asking it some other way is better still? Without an answer to this question, it's unclear to me whether we should talk about an "unpacking fallacy" or a "failure to unpack fallacy".

Comment author: lukeprog 12 January 2014 12:43:17AM 0 points [-]

Here's a handy example discussion of related conjunction issues from the Project Cyclops report:

We have outlined the development of technologically competent life on Earth as a succession of steps to each of [which] we must assign an a priori probability less than unity. The probability of the entire sequence occurring is the product of the individual (conditional) probabilities. As we study the chain of events in greater detail we may become aware of more and more apparently independent or only slightly correlated steps. As this happens, the a priori probability of the entire sequence approaches zero, and we are apt to conclude that, although life indeed exists here, the probability of its occurrence elsewhere is vanishingly small.

The trouble with this reasoning is that it neglects alternate routes that converge to the same (or almost the same) end result. We are reminded of the old proof that everyone has only an infinitesimal chance of existing. One must assign a fairly small probability to one's parents and all one's grandparents and (great)^n-grandparents having met and mated. Also one must assign a probability on the order of 2^-46 to the exact pairing of chromosomes arising from any particular mating. When the probabilities of all these independent events that led to a particular person are multiplied, the result quickly approaches zero. This is all true. Yet here we all are. The [explanation] is that, if an entirely different set of matings and fertilizations had occurred, none of "us" would exist, but a statistically indistinguishable generation would have been born, and life would have gone on much the same.