In light of the portions I quoted from Armstrong and Green's paper, I'll look at Gavin Schmidt's post:

Principle 1: When moving into a new field, don’t assume you know everything about it because you read a review and none of the primary literature.

Score: -2 G+A appear to have only read one chapter of the IPCC report (Chap 8), and an un-peer reviewed hatchet job on the Stern report. Not a very good start…

The paper does cite many other sources than just the IPCC and the "hatchet job" on the Stern Report, including sources that evaluate climate models and their quality in general. ChrisC notes that the author's fail to cite the ~788 references for the IPCC Chapter 8. The authors claim to have a bibliography on their website that includes the full list of references given to them by all academics who suggested references. Unfortunately, as I noted in my earlier comment, the link to the bibliography from http://www.forecastingprinciples.com/index.php?option=com_content&view=article&id=78&Itemid=107 is broken. This doesn't reflect well on the authors (the site on the whole is a mess, with many broken links). Assuming, however, that the authors had put up the bibliography and that it was available as promised in the paper, this critique seems off the mark (though I'd have to see the bibliography to know for sure).

Principle 2: Talk to people who are doing what you are concerned about.

Score: -2 Of the roughly 20 climate modelling groups in the world, and hundreds of associated researchers, G+A appear to have talked to none of them. Strike 2.

This seems patently false given the contents of the paper as I quoted it, and the list of experts that they sought. In fact, it seems like such a major error that I have no idea how Schmidt could have made it if he'd read the paper. (Perhaps he had a more nuanced critique to offer, e.g., that the authors' survey didn't ask enough questions, or they should have tried harder, or contacted more people. But the critique as offered here smacks of incompetence or malice). [Unless Schmidt was reading an older version of the paper that didn't mention the survey at all. But I doubt that even if he was looking at an old version of the paper, it omitted all references to the survey.]

Principle 3: Be humble. If something initially doesn’t make sense, it is more likely that you’ve mis-understood than the entire field is wrong.

Score: -2 For instance, G+A appear to think that climate models are not tested on ‘out of sample’ data (they gave that a ‘-2′). On the contrary, the models are used for many situations that they were not tuned for, paleo-climate changes (mid Holocene, last glacial maximum, 8.2 kyr event) being a good example. Similarly, model projections for the future have been matched with actual data – for instance, forecasting the effects of Pinatubo ahead of time, or Hansen’s early projections. The amount of ‘out of sample’ testing is actually huge, but the confusion stems from G+A not being aware of what the ‘sample’ data actually consists of (mainly present day climatology). Another example is that G+A appear to think that GCMs use the history of temperature changes to make their projections since they suggest leaving some of it out as a validation. But this is just not so, as we discussed more thoroughly in a recent thread.

First off, retrospective "predictions" of things that people already tacitly know, even though those things aren't explicitly used in tuning the models, are not that reliable.

Secondly, it's possible (and likely) that Armstrong and Green missed some out-of-model tests and validations that had been performed in the climate science arena. While part of this can be laid at their feet, part of it also reflects poor documentation by climate scientists of exactly how they were going about their testing. I did read that IPCC AR4 chapter that Armstrong and Green did, and I found it quite unclear on the forecasting side of things (compared to other papers I've read that judge forecast skill, in weather and short-term climate forecasting, macroeconomic forecasting, and business forecasting). This is similar to the sloppy code problem.

Thirdly, the climate scentists whom Armstrong and Green attempted to engage could have been more engaging (not Gavin Schmidt's fault; he wasn't included in the list, and the response rate appears to have been low from mainstream scientists as well as skeptics, so it's not just a problem of the climate science mainstream).

Overall, I'd like to know more details of the survey responses and Armstrong and Green's methodology, and it would be good if they combined their proclaimed commitment to openness with actually having working links on their websites. But Schmidt's critique doesn't reflect too well on him, even if Armstrong and Green were wrong.

Now, to ChrisC's comment:

Call me crazy, but in my field of meteorology, we would never head to popular literature, much less the figgin internet, in order to evaluate the state of the art in science. You head to the scientific literature first and foremost. Since meteorology and climatology are not that different, I would struggle to see why it would be any different.

The authors also seem to put a large weight on “forecasting principles” developed in different fields. While there may be some valuable advice, and cross-field cooperation is to be encouraged, one should not assume that techniques developed in say, econometrics, port directly into climate science.

The authors also make much of a wild goose chase on google for sites matching their specific phrases, such as “global warming” AND “forecast principles”. I’m not sure what a lack of web sites would prove. They also seem to have skiped most of the literature cited in AR4 ch. 8 on model validation and climatology predictions.

Part of the authors' criticism was that the climate science mainstream hadn't paid enough attention to forecasting, or to formal evaluations of forecasting. So it's natural that they didn't find enough mainstream stuff to cite that was directly relevant to the questions at hand for them.

As for the Google search and Google Scholar search, these are standard tools for initiating an inquiry. I know, I've done it, and so has everybody else. It would be damning if the authors had relied only on such searches. But they surveyed climate scientists and worked their way through the IPCC Working Group Report. This may have been far short of full due diligence, but it isn't anywhere near as sloppy as Gavin Schmidt and ChrisC make it sound.

On Critique #1:

Since you are using Real Climate and Skeptical Science as sources, did you read what they had to say about the Armstrong and Green paper and about Nate Silver's chapter?

Gavin Schmidt's post was short, funny but rude; however ChrisC's comment looks much more damning if true. Is it true?

Here is Skeptical Science on Nate Silver. It seems the main cause of error in Hansen's early 1988 forecast was an assumed climate sensitivity greater than that of the more recent models and calculations (4.2 degrees rather than 3 degrees). Whereas IPCC's 1990 forecast had problems predicting the inputs to global warming (amount of emissions, or radiative forcing for given emissions) rather than the outputs (resulting warming). Redoing accounting for these factors removes nearly all the discrepancy.

Hmmmm. (ETA: following claim is incorrect) They're judging that the planets are rocky by measuring their mass, not by noticing that they're actually rocky.

If you don't have a Jupiter-sized core out there sucking up all the gas, why would gas planets need to end up as giants? They naturally could do that - that happened with the star, after all, but it doesn't seem inevitable to me, and it might not even be common.

In that case, the earth-mass planets would be gas planets after all. If you think this is a stretch, keep in mind that these are specifically in systems noted to be low metallicity. Suggesting that they might not be high in metals after all is not much of a stretch.

I don't get the reason panspermia needs to be involved. Simply having a minimum metallicity threshold for getting started would do the job.

Hmmm... I'll have a go. One response is that the "fully general counter argument" is a true counter argument. You just used a clever rhetorical trick to stop us noticing that.

If what you are calling "efficiency" is not working for you, then you are - ahem - just not being very efficient! More revealingly, you have become fixated on the "forms" of efficiency (the metrics and tick boxes) and have lost track of the substance (adopting methods which take you closer to your true goals, rather than away from them). So you have steelmanned a criticism of formal efficiency, but not of actual efficiency.

As I mentioned in the post:

I read many papers from a diverse array of sources. I arrived at most papers either by clicking links on one of the blogs or websites mentioned above, or using Google or Google Scholar searches for specific topics. Any paper that I use as input to my opinion in a specific post will be explicitly linked in that post.

I can't include a list of papers right now because the list of papers will itself be determined in real time during my inquiries, but I will link to the ones I reference at each stage of research.

Of course a number of the blogs convey the message that climate science as a whole is utterly partisan and biased, so any output of climate scientists through secondary sources and summaries is untrustworthy. If you can't analyse the underlying primary evidence, and do not assign negligible prior probability to such a mass scientific conspiracy (or mass scientific error) then it is hard to refute that mindset. But you still have to ask who has the greater incentives here: is it really poorly paid scientists pushing a conspiracy or collective fantasy to get a bit more funding, or is it highly paid lobbyists, firms and commentators defending a trillion dollar industry, one which would be doomed by serious action on climate change?

This needs to be unpacked.

poorly paid scientists pushing a conspiracy or collective fantasy to get a bit more funding

Scientists are driven by a range of motives including research prestige and ego (those who have made statements in the past want to have those statements vindicated), the desire to impress and influence peers, etc. Getting more funding is part of the status game. These incentives can distort their findings, not necessarily forever but for long enough (just like the climate system, the system of scientific discovery is not in equilibrium; there are lags).

More importantly, simple cognitive and human biases can get in the way of proper analysis. One of the points I raise is that it's possible, based on Armstrong and Green's critique (I'm still investigating this) that climate scientists appear not to have consulted people in other relevant areas of expertise (specifically, forecasting and statistics). People in these areas of expertise have come up with a lot of relevant and counter-intuitive findings about how to go about this sort of tricky data analysis.

highly paid lobbyists, firms and commentators defending a trillion dollar industry

Of the sources I list, which ones do you think is written or funded by people who fit this description?

I believe that Real Climate and Skeptical Science are largely authored by working climate scientists

Judith Curry is also a working climate scientist.

Stephen McIntyre isn't a working climate scientist, but his criticism of Mann's statistical errors (which aren't necessarily relevant to the main arguments for AGW) have been acknowledged as essentially correct. I also took a reasonably detailed look at the specifics of the argument, and although I can't have very high confidence, I'm inclined to believe that McIntyre was right. He seems to be sufficiently rigorous in his work and a sufficiently strong skeptic that his critiques are worth reading. Moreover, he rarely claims more confidence than is warranted: he isn't publishing his own theories of climate change, he sees it as his job mainly to catch bad statistics.

Watts Up With That? seems to have poorer quality control in general, and often publishes opinion-type posts, but many of those opinion posts are guest posts by working climate scientists (albeit not necessarily high-profile ones). i wouldn't assign high credence to the blog.

As I mentioned in the post:

I read many papers from a diverse array of sources. I arrived at most papers either by clicking links on one of the blogs or websites mentioned above, or using Google or Google Scholar searches for specific topics. Any paper that I use as input to my opinion in a specific post will be explicitly linked in that post.

I can't include a list of papers right now because the list of papers will itself be determined in real time during my inquiries, but I will link to the ones I reference at each stage of research.

Of course a number of the blogs convey the message that climate science as a whole is utterly partisan and biased, so any output of climate scientists through secondary sources and summaries is untrustworthy. If you can't analyse the underlying primary evidence, and do not assign negligible prior probability to such a mass scientific conspiracy (or mass scientific error) then it is hard to refute that mindset. But you still have to ask who has the greater incentives here: is it really poorly paid scientists pushing a conspiracy or collective fantasy to get a bit more funding, or is it highly paid lobbyists, firms and commentators defending a trillion dollar industry, one which would be doomed by serious action on climate change?

This needs to be unpacked.

poorly paid scientists pushing a conspiracy or collective fantasy to get a bit more funding

Scientists are driven by a range of motives including research prestige and ego (those who have made statements in the past want to have those statements vindicated), the desire to impress and influence peers, etc. Getting more funding is part of the status game. These incentives can distort their findings, not necessarily forever but for long enough (just like the climate system, the system of scientific discovery is not in equilibrium; there are lags).

More importantly, simple cognitive and human biases can get in the way of proper analysis. One of the points I raise is that it's possible, based on Armstrong and Green's critique (I'm still investigating this) that climate scientists appear not to have consulted people in other relevant areas of expertise (specifically, forecasting and statistics). People in these areas of expertise have come up with a lot of relevant and counter-intuitive findings about how to go about this sort of tricky data analysis.

highly paid lobbyists, firms and commentators defending a trillion dollar industry

Of the sources I list, which ones do you think is written or funded by people who fit this description?

I believe that Real Climate and Skeptical Science are largely authored by working climate scientists

Judith Curry is also a working climate scientist.

Stephen McIntyre isn't a working climate scientist, but his criticism of Mann's statistical errors (which aren't necessarily relevant to the main arguments for AGW) have been acknowledged as essentially correct. I also took a reasonably detailed look at the specifics of the argument, and although I can't have very high confidence, I'm inclined to believe that McIntyre was right. He seems to be sufficiently rigorous in his work and a sufficiently strong skeptic that his critiques are worth reading. Moreover, he rarely claims more confidence than is warranted: he isn't publishing his own theories of climate change, he sees it as his job mainly to catch bad statistics.

Watts Up With That? seems to have poorer quality control in general, and often publishes opinion-type posts, but many of those opinion posts are guest posts by working climate scientists (albeit not necessarily high-profile ones). i wouldn't assign high credence to the blog.

I've noticed that you've listed a lot of secondary sources (books, blogs, IPCC summaries) but not primary sources (published papers by scientists in peer-reviewed journals). Is there a reason for this e.g. that you do not have access to the primary sources, or find them indigestible?

If you do need to rely on secondary sources, I'd suggest to focus on books and blogs whose authors are also producing the primary sources. Of the blogs you mention, I believe that Real Climate and Skeptical Science are largely authored by working climate scientists, whereas the others are not.

Of course a number of the blogs convey the message that climate science as a whole is utterly partisan and biased, so any output of climate scientists through secondary sources and summaries is untrustworthy. If you can't analyse the underlying primary evidence, and do not assign negligible prior probability to such a mass scientific conspiracy (or mass scientific error) then it is hard to refute that mindset. But you still have to ask who has the greater incentives here: is it really poorly paid scientists pushing a conspiracy or collective fantasy to get a bit more funding, or is it highly paid lobbyists, firms and commentators defending a trillion dollar industry, one which would be doomed by serious action on climate change?

Or why stage 1 is so long.

This all looks clever, apart from the fact that the AI becomes completely indifferent to arbitrary changes in its value system. The way you describe it, the AI will happily and uncomplainingly accept a switch from a friendly v (such as promoting human survival, welfare and settlement of Galaxy) to an almost arbitrary w (such as making paperclips), just by pushing the right "update" buttons. An immediate worry is about who will be in charge of the update routine, and what happens if they are corrupt or make a mistake: if the AI is friendly, then it had better worry about this as well.

Interestingly, the examples you started with suggested that the AI should be rewarded somehow in its current utility v as a compensation for accepting a change to a different utility w. That does sound more natural, and more stable against rogue updates.