Can someone explain to me why this argument is considered interesting or good? I genuinely do not see the merit in it.
It takes a very valid concern ("don't use your models beyond their domain of applicability") which smart people are likely to agree with, and generalizes it to make an argument against reductionism. In order to realize what's wrong with the argument, you need to be aware of concepts such as privileging the hypothesis, which many people aren't. At the same time, it also appeals to people's desire to not be extremists and take the middle ground. The combination of those two factors makes it very compelling for a certain kind of mindset.
Concretizing the abstract is an interesting blog post in that it makes a relatively cogent argument for non-reductionism. While I don't agree with it, I found it useful in that it helped me better understand how intelligent non-reductionists think. It also helped clarify to me an old distinction, that of philosophers versus engineers.
I find this interesting in the way that smart people are likely to disagree with the correct interpretation of some of its claims - while others would say the post is worshipping the mysterious, others would say that it's just making reasonable cautions about the inherent methodological limitations of a certain approach. One might even think that it's essentially making a similar point as Eliezer's warning about floating beliefs, and therefore to agree with the Sequences. The caution of "beware of thinking that your abstractions say everything that there is to be said about something" is a reasonable one, and people do clearly make that mistake sometimes.
I expect that part of what influences how plausible one finds this argument depends on whether one has more of an "engineer's mindset" or a "philosopher's mindset". Somebody with an engineer's mindset will think that "yes, the abstractions we use might be imperfect, but what else do you propose we use? They're still the best tool for accomplishing stuff, and anything else is just philosophcial nonsense that isn't grounded in anything". Whereas the philosopher is less interested in using their knowledge to "accomplish stuff", and more interested in the ideas and their implications themselves.
As an aside, this distinction might be part of the reason why we have so many computer or hard science folks on this site. Partially it's because Eliezer used a lot of CS jargon in writing the Sequences, but probably also because the Sequences, while philosophical in nature, are also very focused on practical results and getting empirical predictions out of your beliefs.
Looking at what we could use this distinction for (and thus taking an engineer's mindset) some people here have mentioned getting an "ick" reaction from religious people, just due to those people having strong false beliefs. I think that, combined with properly understanding the emotional basis of religion, an understanding of the philosopher / engineer distinction can help avoid that reaction. Our values determine our beliefs, and there are plenty of religious people who aren't stupid, crazy, or anything like that. They might simply be philosophers instead of engineers, or they might be engineers who are more interested in the instrumental benefits of religion than the rather marginal benefits of x-rationality. (Amusingly, such a "religious engineer" might justifiably consider our obsession with "truth" as just an odd philosophical pursuit.)