Unnamed comments on You're Entitled to Arguments, But Not (That Particular) Proof - Less Wrong

57 Post author: Eliezer_Yudkowsky 15 February 2010 07:58AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (221)

You are viewing a single comment's thread. Show more comments above.

Comment author: SilasBarta 15 February 2010 04:45:42PM *  19 points [-]

I was wondering how long it would be until the AGW issue was directly broached on a top-level post. Here I will state my views on it.

First, I want to fend off the potential charge of motivated cognition. I have spent the better part of two years criticizing fellow "libertarians" for trivializing the issue, and especially for their rationalizations of "Screw the Bengalis" even when they condition on AGW being true. I don't have the links gathered in one place, but just look here and here, and linked discussions, for examples.

That said, here are the warning signs for me (this is just to summarize, will gather links later if necessary):

1) Failed predictions. Given the complexity of the topic, your models inevitably end up doing curve-fitting. (Contrary to a popular misconception, they do not go straight from "the equations they design planes from" to climate models.) That gives you significant leeway in fitting the data to your theory. To be scientific and therefore remove the ability of humans to bias the data, it is vital that model predictions be validated against real-world results. They've failed, badly: they predicted, by existing measures of "global temperature", that it would be much higher than it is now.

2) Anti-Bayesian methodology accepted as commonplace. As an example, regarding the "hide the decline" issue with the tree rings, here's what happened: Scientists want to know how hot it was millenia ago. Temperature records weren't kept then. So, they measure by proxies. One common proxy is believed to be tree rings. But tree rings don't match the time period in which we have the best data.

The correct procedure at this point is to either a) recognize that they aren't good proxies, or b) include them in toto as an outlier data point. Instead, what they do is to keep all the data points that support the theory, and throw out the rest, calling it a "divergence problem", and further, claim the remaining points as additional substantiation of the theory. Do I need to explain here what's wrong with that?

And yet the field completely lacks journals with articles criticizing this.

3) Error cascades. Despite the supposed independence of the datasets, they ultimately come from only a few interbred sources, and further data is tuned so that it matches these data sets. People are kept out of publication, specifically on the basis that their data contradicts the "correct" data.

Finally, you can't just argue, "The scientists believe AGW, I trust scientists, ergo, the evidence favors AGW." Science is a method, not a person. AGW is credible to the exent that there is Bayesian evidence for it, and to the extent scientists are following science and finding Bayesian evidence. The history of the field is a history of fitting the data to the theory and increasing pressure to make sure your data conforms to what the high-status people decreed is correct.

Again, if the field is cleansed and audited and the theory turns out to hold up and be a severe problem, I would love for CO2 emissions to finally have their damage priced in so that they're not wastefully done, and I pity the fools that demand Bengalis go and sue each emitter if they want compensation. But that's not where we are.

And I don't think it's logically rude to demand that the evidence adhere to the standard safeguards against human failings.

Comment author: Unnamed 16 February 2010 01:51:04AM 2 points [-]

How thorough is your knowledge of the AGW literature, Silas? I'm only familiar with bits and pieces of it, much of it filtered through sties like Real Climate, but what I've seen suggests that climate scientists are doing better than you indicate. For instance, the paper described here includes estimates excluding tree ring data as well as estimates that include tree ring data, because of questions about the reliability of that data (and it cites a bunch of other articles that have addressed that issue). They also describe methods for calibrating and validating proxy data that I haven't tried to understand, but which seem like the sort of thing that they should be doing.

Comment author: brazil84 17 February 2010 12:34:50AM 1 point [-]

I think the narrow issue of multi-proxy studies teaches an interesting lesson to folks who like to think of things in terms of Bayesian probabilities.

I would submit that at a bare minimum, any multi-proxy study (such as the one you cite) needs to provide clear inclusion and exclusion criteria for the proxies which are used and not used.

Let's suppose that there is a universe of 300 possible temperature proxies which can be used and Michael Mann chooses 30 for his paper. If he does not explain to us how he chose those 30, then how can anyone have any confidence in his results?

I haven't read the paper myself, but here's what the infamous Steve McIntyre says:

I identified 33 non-tree ring proxies with that started on or before 1000 – many, perhaps even most, of these proxies are new to the recon world. How were these particular proxies selected? How many proxies were screened prior to establishing this network? Mann didn’t say

Comment author: SilasBarta 16 February 2010 04:49:44PM *  -1 points [-]

Yes, I've followed Real Climate, on and off, and with greater intensity after the Freakonomics fiasco (where RCers were right because of how sloppy the Freakons were), which directly preceded climategate. FWIW, I haven't been impressed with how they handle stuff outside their expertise, like the time-discounting issue.

As for the paper you mention, my primary concern is not that the tree data by itself overturns everything, but rather, that they consider it a valid method to clip out disconfirmatory data while still counting the remainder as confirmatory, which makes me wonder how competent the rest of the field is.

The responses on RC about the tree ring issue reek of "missing the point":

The paper in question is the Mann, Bradley and Hughes (1998) Nature paper on the original multiproxy temperature reconstruction, and the ‘trick’ is just to plot the instrumental records along with reconstruction so that the context of the recent warming is clear. Scientists often use the term “trick” to refer to a “a good way to deal with a problem”, rather than something that is “secret”, and so there is nothing problematic in this at all. As for the ‘decline’, it is well known that Keith Briffa’s maximum latewood tree ring density proxy diverges from the temperature records after 1960 (this is more commonly known as the “divergence problem”–see e.g. the recent discussion in this paper) and has been discussed in the literature since Briffa et al in Nature in 1998 (Nature, 391, 678-682). Those authors have always recommend not using the post 1960 part of their reconstruction, and so while ‘hiding’ is probably a poor choice of words (since it is ‘hidden’ in plain sight), not using the data in the plot is completely appropriate, as is further research to understand why this happens.

Not using the data at all would be appropriate (or maybe not, since you should include disconfirmatory data points). Including only the data points that agree with you would be very inappropriate, as they certainly can't count as additional proof once they're filtered for agreement with the theory.

Comment author: Unnamed 16 February 2010 07:20:20PM 1 point [-]

I'm growing less clear about what your complaint is. If you're just pointing out a methodological problem in that one paper then I agree with you. If you're claiming that the whole field is so messed up that no one even realizes it's a problem, then the paper that I linked looks like a counterexample to your claim. The authors seem to recognize that it's bad to make ad hoc choices about which proxies to use or which years to apply them to, so they came up with a systematic procedure for selecting proxies (it looks similar to taking all of every proxy that correlates significantly with the 150 years of instrumental temperature records and then averaging those proxy estimates together, but more complicated). And because tree-ring data had been the most problematic (in having a poor fit with the temperature record), they ran a separate set of analyses that excluded those data. They may not explicitly criticize the other methodology, but they're replacing it with a better methodology, which is good enough for me.

Comment author: SilasBarta 16 February 2010 10:41:54PM 2 points [-]

You don't understand why I'm suspicious that a fundamental problem with their methodology, widely used as proof, is only being rooted out in 2008?

Comment author: Eliezer_Yudkowsky 16 February 2010 10:52:54PM 3 points [-]

Be glad it's happening at all.

Comment author: Unnamed 17 February 2010 05:20:25AM 1 point [-]

Is it only being rooted out in 2008? There have been a bunch of different proxy reconstructions over the years - are you saying that this 2008 paper was the first one to avoid that methodological problem? Do you know the climate literature well enough to be making these kinds of statements?

Comment author: Luke_A_Somers 10 December 2011 06:12:50PM 0 points [-]

There are several factors that can limit] tree growth. Sometimes, low temperature is the bottleneck. So, the tree ring data can in any case be considered a reliable indicator of a floor on the temperature. It isn't any colder than this point.

They try to pick trees that are more likely to find low temperature the bottleneck. Sometimes it isn't.

That doesn't mean that the whole series is useless, even if they happen to be using it wrong (and I don't know that they are).