It's a bit long, sorry.

Let me bore y'all with a bit of my puzzle-solving MoR.

Near the end of 2010 I came up with a theory (which I codenamed NBT), some far-out thing that explained a few observations, and removed a good deal of confusion. As an aside it also led me to predict the existence of a certain clue - more on that later. Also, I think it may be endgame-relevant.

Since this theory was kinda subtle, I felt I needed more data on it, so I filed it away and waited. Somewhere along the way I realized that it explained even more things than I thought, thus raising my confidence in it (retrograde prediction? - my surprise about it is evidence to me, even if it wouldn't convince anyone else), though of course it could still be subject to positive bias, as I was well aware.

Enter the final batch of SA arc, late August and early September (2011). It contained an unlikely event, one I would have bet against seeing, without NBT. With it, it was just the kind of thing I would expect. So I marked NBT confirmed. Shortly afterwards a random re-reading of an older part led me to discover the clue I predicted earlier, an innocent little sentence I completely missed before, one that contains a hint connected to the nature of magic. As it happens, this clue also explains how the Interdict of Merlin was made - the mechanism.

Thus feeling my theory NBT strongly confirmed I confronted the one part I was still confused about, and within ten minutes I moved from 'a bit tricky but possible' to 'simple, elegant, neat'. I didn't need any knowledge that wasn't available when I came up with NBT, so I could have had that answer a lot earlier - I'll call the crucial bit of understanding V-factor.

(Also, Self Actualization confirmed one more prediction of mine. I suppose for others that particular bit of noticing-your-confusion could be a hint for a difference-from-canon mystery that didn't lead anywhere; for me, it was evidence I was right all along about one more thing. This theory was required for later, but I'm sure I wouldn't have needed the confirmation; I was pretty sure of this one. Just mentioning that to show why I'm glad SA happened.)

All this reminded me of the moral of That Alien Message, specifically how limiting it is to think 'we need more information' and do nothing, instead of actually sitting down, and confronting the mystery. On one hand I saw things most would say can't possibly be seen, on the other hand I failed to find out things fast enough because I didn't think I could possibly have seen them.

Because I'm not perfect (yet), I didn't update on everything immediately; to be more precise, it wasn't until January of 2012 that I applied the V-factor to my previous theories of the nature of MoR magic, but when I did, I became unconfused. The First Cause of Verresverse Magic became clear. Looking back, and seeing that I could have had the whole thing a year earlier if I tried - not hindsight, the conclusions that led to it are clearly traceable - really drove the point home.

My point with all this? Methods of Rationality is full of subtle puzzles, reasons beneath the obvious, as others have pointed out in much less word, and also plenty of ways to practice and test your rationality skills. Noticing confusion, generating alternative hypotheses, countering positive bias, above all. And yes, these are well constructed puzzles, not like the messy reality, but the point is to have the grooves worn into the mind.

Nitpicking the story, complaining about things we wouldn't even notice from other authors, holding Eliezer to higher standards, because 'Hey, we're rational(ists)!' while failing to notice that these may well be deliberate hints of something deeper ... it's ... well ... kinda stupid, ain't it? Like how sophisticated fan-fiction readers might pattern-match Harry of early chapters as Mary Sue, while failing to notice all the clues about his coldness, concluding it is a sign of incompetent author, when it is in fact a hint of something more going on. A useful lesson about how noticing confusion and generating an alternative hypothesis might be needed, at best.

Also, if you have been reading LW for a while and have been passionate about improving your rationality, yet you failed to notice so far how good an opportunity MoR is ... I just spoiled it for you; it was a test, and you failed it. Sorry.