I think the test for EY's articles aren't predictions he makes, which I don't recall many of, but what you can do with the insight he offers. For example, does it allow you to deal with your own problems better? Has it improved your instrumental rationality, including your epistemic rationality?

best track record of any top pundit in the US

The study linked to meant next to nothing in my eyes. It studied political predictions in an election year by political hacks on tv. 2007-2008. Guess what? IN an election cycle that liberals beat conservatives, the liberal predictions more often came true than conservative predictions.

Reminds me of the reported models of mortgage securities, created using data from boom times only.

Krugman was competing with a bunch of other political hacks and columnists. I doubt that accuracy is the highest motivation for any of them. The political hacks want to curry support, and the columnists want to be invited on tv and have their articles read. I'd put at least 3 motivations above accuracy for that crowd: manipulate attitudes, throw red meat to their natural markets, and entertain. It's Dark Arts, all the way, all the time.

My understanding is that for the most part SI prefers not to publish the results of their AI research, for reasons akin to those discussed here. However they have published on decision theory, presumably because it seems safer than publishing on other stuff and they're interested in attracting people with technical chops to work on FAI:

http://singinst.org/blog/2010/11/12/timeless-decision-theory-paper-released/

I would guess EY sees himself as more of a researcher than a forecaster, so you shouldn't be surprised if he doesn't make as many predictions as Paul Krugman.

Also, here's a quote from his paper on cognitive biases affecting judgment of global risks:

Once upon a time I made up overly detailed scenarios, without realizing that every additional detail was an extra burden. Once upon a time I really did think that I could say there was a ninety percent chance of Artificial Intelligence being developed between 2005 and 2025, with the peak in 2018. This statement now seems to me like complete gibberish. Why did I ever think I could generate a tight probability distribution over a problem like that? Where did I even get those numbers in the first place?

So he wasn't born a rationalist. (I've been critical of him in the past, but I give him a lot of credit for realizing the importance of cognitive biases for what he was doing and popularizing them for such a wide audience.) My understanding was that one of the primary purposes of the sequences was to get people to realize the importance of cognitive biases at a younger age than he did.

Obviously I don't speak for SI or Eliezer, so take this with a grain of salt.

What does a decision theory that "has a solid theoretical background but turns out to be terrible" look like when implemented?

My understanding is that for the most part SI prefers not to publish the results of their AI research, for reasons akin to those discussed here. However they have published on decision theory, presumably because it seems safer than publishing on other stuff and they're interested in attracting people with technical chops to work on FAI:

http://singinst.org/blog/2010/11/12/timeless-decision-theory-paper-released/

I would guess EY sees himself as more of a researcher than a forecaster, so you shouldn't be surprised if he doesn't make as many predictions as Paul Krugman.

Also, here's a quote from his paper on cognitive biases affecting judgment of global risks:

Once upon a time I made up overly detailed scenarios, without realizing that every additional detail was an extra burden. Once upon a time I really did think that I could say there was a ninety percent chance of Artificial Intelligence being developed between 2005 and 2025, with the peak in 2018. This statement now seems to me like complete gibberish. Why did I ever think I could generate a tight probability distribution over a problem like that? Where did I even get those numbers in the first place?

So he wasn't born a rationalist. (I've been critical of him in the past, but I give him a lot of credit for realizing the importance of cognitive biases for what he was doing and popularizing them for such a wide audience.) My understanding was that one of the primary purposes of the sequences was to get people to realize the importance of cognitive biases at a younger age than he did.

Obviously I don't speak for SI or Eliezer, so take this with a grain of salt.

On quantum mechanics: the many-worlds interpretation probably shouldn't be referred to as "his" interpretation (around before he was born, etc.), and there's been some experimental work defending it (for example, entangling 10^4 particles and not seeing a certain kind of collapse), bit it's not very strong evidence.

He also made a lot of super-wrong plans/predictions over a decade ago. But on the other hand, that was a while ago.

On a related note... has Eliezer successfully predicted anything? I'd like to see his beliefs pay rent, so to speak. Has his interpretation of quantum mechanics predicted any phenomena which have since been observed? Has his understanding of computer science and AI lead him to accurately predict milestones in the field before they have happened?