Scholarship: how to tell good advice from bad advice?

ChrisHallquist

Lukeprog has done some great posts talking about how scholarship has benefited him personally and how to do it. However, he hasn't talked much about how to tell the good from the bad, and I think this is an especially big problem with it comes to "how to"s, rather than standard academic subjects.

The thing is this: writing on academic subjects tends to be dominated by academics seeking to gain status from their work. Not all fields have a strong correlation between status and making actual discoveries, but many (perhaps most) do. So for academic subjects, the big challenge when it comes to making sure you're not reading nonsense is to try to evaluate the soundness of the field in question.

When it comes to seeking out "how to" advice, though, writing on the subjects seems to be dominated by people looking to make money selling advice. That means that if bad advice will sell better than good advice, they'll give the bad advice. Even if their writing includes legitimate advice, they have incentives to give a distorted picture of the subject they're writing on if it will sell better (for example, making whatever they're talking about sound easier than it really is). In some cases, advice books end up trying to nudge you towards buying some more expensive thing the author is selling.

So: when you're researching a "how to" topic, how do you tell the good advice from the bad advice?

When you receive advice, it must fall into one of these categories:

The advice is worthless or even outright bad.
The advice is valuable, but it's common knowledge, or at least can be obtained from public information in a straightforward way.
The advice is valuable, and it's not common knowledge, nor can it be obtained from public information in any straightforward way.

Now, if you believe you're receiving advice that is in category (3), you must ask yourself what makes you so special that you are privy to this information. This leads to the following heuristics:

If the advice is in category (2), it is likely to be good if the source is reputable. For example, a book or website about programming written by a reputable author is likely to give you good advice on how to program. (But note that while it certainly adds value in terms of a convenient and attractive presentation, such a source doesn't give any significant information that wouldn't be available from other public sources.)
If the advice purports to be in category (3), and yet the source of information is public (e.g. a book or website, even a non-free one), it's almost certainly bunk. The only exception is if the message is highly unpopular or counter-intuitive, and somehow you know that you have overcome biases that prevent most people from evaluating it correctly, which is very difficult and rare. For example, nearly any book that claims to bring special wisdom about investment, career, relationships, etc. is bunk.
If the advice comes from a person on a private occasion, then there are several steps that you need to do. First, does this person show clear indications of the relevant knowledge and competence? If not, it's likely bunk for obvious reasons. Second, is the advice in category (2) or (3)? If it's (2), it's probably good, though it still pays off to check against other sources of information. If it purports to be (3), then you need to do the third, and most difficult evaluation: does this person have the motivation for an extraordinary degree of altruism towards you? If not, it's likely bunk, or otherwise they wouldn't grant you this privilege. If yes, for example if you're getting advice from your parents, then it is probably highly valuable.

One common failure mode is when people believe they're giving you advice of type (3), but in reality, their motivation for altruism towards you is weaker than their motivation for saying things that have high signaling value (and omitting things that have negative such value). This is one danger of socializing with people who are higher-status and more accomplished than you -- you'll be tempted to take their advice seriously, but in reality, even if they are giving it with good intentions, it's likely to be heavily censored and distorted so as to maximize its signaling value.

(This is exacerbated by the fact that good no-nonsense advice on topics that involve any aspects of human social behavior, both personal and professional, tends to sound crass, disreputable, cynical, or worse.)

This is true, but a simplification. Specifically, it doesn't distinguish cases where good advice is mostly impersonal (e.g. how to invest money) from cases where the best advice will be highly personalized (e.g. diet).

In many complicated fields, like diet, good advice needs to be individual. Learning enough about the field to choose the right advice yourself may take years. And it's prohibitively expensive to find what works best for you by trying everything. At best, you'll stick with the first thing that works moderately well.

So I propose category (4): t... (read more)

14asr14y

Hrm. There's a category 2.5 of "advice which is common knowledge to long-standing members of a small subculture but not to the public and that isn't written down." These are cases where the information isn't secret, but is primarily conveyed word-of-mouth, not in print, and where it's too specialized to be "conventional wisdom". It might be that there's some piece of advice that most expert, say, patent lawyers would give you, but that wouldn't be in a standard book because the topic is too narrow or esoteric. In my case, I've gotten valuable career advice from senior members of my profession. I don't think it was a unique boon to me and their advice parallels conventional wisdom, but with details that are specific to my narrow field.

30sixes_and_sevens14y

There's a type of advice I've observed which I'm having trouble categorising into the above: advice which is valuable, but which has a prerequisite level of competence or understanding before you can use it. I've been swing dancing for about four and a half years, and I've taken a lot of classes and workshops in it. There are several common pieces of advice that get thrown around by teachers: "keep your feet under you", "dance into the floor", "engage your core"... they're generally referring to how something feels when you're doing it, which has a reasonable margin for subjectivity, so some people will hear advice like that and think "oh, yeah, that makes sense", while others will hear it and think "well where else would my feet be? Over me?" Good instructors find a way to give you the bodily experience without giving you misinterpretable verbal advice. "Engage your core" can mean different things to different people. There's a reasonable amount of crossover between the swing dance community and the circus skills community, and when an aerial acrobat hears "engage your core", they tense up like they're about to be thrown, which is not what's meant in the dance context. But if someone says "imagine you've got a sword stuck point-first in your navel, and you have to keep that sword horizontal", that's quite a specific piece of advice which directly addresses what is meant. Sometimes you're just not ready for a piece of advice. Earlier this year at a weekend-long event, a well-respected and reputable instructor said something in a workshop. I can't remember his exact wording, but in my notes for the class I wrote "the tension and compression you feel in your hands is a consequence of your bodies moving, not a cause". I thought about this over the rest of the weekend and it completely blew my mind and made me revise huge chunks of how I thought the dance mechanically worked. If I'd heard it two years ago, my response to it would probably have been a lot less product

19

Scholarship: how to tell good advice from bad advice?

19

19

19

Scholarship: how to tell good advice from bad advice?

19

19