It's a bit long, sorry.
Let me bore y'all with a bit of my puzzle-solving MoR.
Near the end of 2010 I came up with a theory (which I codenamed NBT), some far-out thing that explained a few observations, and removed a good deal of confusion. As an aside it also led me to predict the existence of a certain clue - more on that later. Also, I think it may be endgame-relevant.
Since this theory was kinda subtle, I felt I needed more data on it, so I filed it away and waited. Somewhere along the way I realized that it explained even more things than I thought, thus raising my confidence in it (retrograde prediction? - my surprise about it is evidence to me, even if it wouldn't convince anyone else), though of course it could still be subject to positive bias, as I was well aware.
Enter the final batch of SA arc, late August and early September (2011). It contained an unlikely event, one I would have bet against seeing, without NBT. With it, it was just the kind of thing I would expect. So I marked NBT confirmed. Shortly afterwards a random re-reading of an older part led me to discover the clue I predicted earlier, an innocent little sentence I completely missed before, one that contains a hint connected to the nature of magic. As it happens, this clue also explains how the Interdict of Merlin was made - the mechanism.
Thus feeling my theory NBT strongly confirmed I confronted the one part I was still confused about, and within ten minutes I moved from 'a bit tricky but possible' to 'simple, elegant, neat'. I didn't need any knowledge that wasn't available when I came up with NBT, so I could have had that answer a lot earlier - I'll call the crucial bit of understanding V-factor.
(Also, Self Actualization confirmed one more prediction of mine. I suppose for others that particular bit of noticing-your-confusion could be a hint for a difference-from-canon mystery that didn't lead anywhere; for me, it was evidence I was right all along about one more thing. This theory was required for later, but I'm sure I wouldn't have needed the confirmation; I was pretty sure of this one. Just mentioning that to show why I'm glad SA happened.)
All this reminded me of the moral of That Alien Message, specifically how limiting it is to think 'we need more information' and do nothing, instead of actually sitting down, and confronting the mystery. On one hand I saw things most would say can't possibly be seen, on the other hand I failed to find out things fast enough because I didn't think I could possibly have seen them.
Because I'm not perfect (yet), I didn't update on everything immediately; to be more precise, it wasn't until January of 2012 that I applied the V-factor to my previous theories of the nature of MoR magic, but when I did, I became unconfused. The First Cause of Verresverse Magic became clear. Looking back, and seeing that I could have had the whole thing a year earlier if I tried - not hindsight, the conclusions that led to it are clearly traceable - really drove the point home.
My point with all this? Methods of Rationality is full of subtle puzzles, reasons beneath the obvious, as others have pointed out in much less word, and also plenty of ways to practice and test your rationality skills. Noticing confusion, generating alternative hypotheses, countering positive bias, above all. And yes, these are well constructed puzzles, not like the messy reality, but the point is to have the grooves worn into the mind.
Nitpicking the story, complaining about things we wouldn't even notice from other authors, holding Eliezer to higher standards, because 'Hey, we're rational(ists)!' while failing to notice that these may well be deliberate hints of something deeper ... it's ... well ... kinda stupid, ain't it? Like how sophisticated fan-fiction readers might pattern-match Harry of early chapters as Mary Sue, while failing to notice all the clues about his coldness, concluding it is a sign of incompetent author, when it is in fact a hint of something more going on. A useful lesson about how noticing confusion and generating an alternative hypothesis might be needed, at best.
Also, if you have been reading LW for a while and have been passionate about improving your rationality, yet you failed to notice so far how good an opportunity MoR is ... I just spoiled it for you; it was a test, and you failed it. Sorry.
Nobody's infallible. What kind of rationalist approaches a work with a couple holes in it and leaves under the assumption that the writer is simply so perfect that everything will be addressed and wrapped up neatly by the end? Isn't it far more likely that in writing a fantastic teaching tool/test, Eliezer occasionally leaves a thread or two hanging?
Warning: As per the official spoiler policy, the following discussion may contain unmarked spoilers for up to the current chapter of the Methods of Rationality. Proceed at your own risk.
Assume HPMOR was written by a super-intelligence implementing the CEV of Eliezer Yudkowsky and assorted literary critics. What would it have written differently?
... is what I want to know, but that's hard to answer. So here's an easier question:
In what ways do you think Eliezer's characterisations/world-building/plot-fu are sub-optimal? <optional> How could they be made less sub-optimal? </optional>
(My own ideas are in the comments.)
To put it another way... Assume a group of intrepid fanfic writers in the late 2020s are planning to write a reboot. What parts of Eliezer's story do you think they should tweak?
And just to make sure we're all on the same page: Eliezer isn't going to go back and change anything he's written to bring it in line with anything suggested here. This is purely an "Ah, just consider the possibilities!" thread.
... which means that we can safely suggest drastic rewrites encompassing 30 chapters or something. Or change fundamental facts about the world.
(Exercise due restraint on this one. Getting rid of the Ministry/the Noble Houses/blood purism would probably turn the story into something completely different; this isn't what we're trying to do here.)
With that, let the nit-picking begin!!