It is a runaway peacock's-tail phenomenon: Someone made something that stood out in some way, and it got attention; and people learned to like things like that, and so others made things that stood out more in the same way, until we ended up with Alban Berg.

As people learn more about an art form, they can more-easily predict it, and need more and more novelty to keep them interested;

I suspect Methods of Rationality may be the end product of a similar phenomenon with respect to a number of trends in speculative fiction, e.g., of putting in more and more elaborate Xanatos Gambits and more and more subtle pop culture references.

Or as Eliezer put it:

it's hard to beat the Algorithm of Awesome, which works as follows:

First, know the overarching direction in which your fic is going. Then, think of possible events that move in this direction. If they are awesome, add them to the plot. If they are not awesome, leave them out.

Try looking at the above quote while tabooing the word "awesome", or better yet replace it with a word that has a similar meaning to an art movement you aren't involved in e.g., "groovy" for psychedelic, "transgresive" for modern art, etc.