FAWS comments on "Trials and Errors: Why Science Is Failing Us" - Less Wrong

7 Post author: gwern 19 December 2011 06:48PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (20)

You are viewing a single comment's thread. Show more comments above.

Comment author: Manfred 19 December 2011 07:44:02PM 5 points [-]

Huh, I was expecting Science to mean "science as a social institution," but he really is making the strong claim that science as a way of learning things is "failing us" because the human body is complicated. Where of course the problem is, "failing us relative to what?"

Comment author: FAWS 19 December 2011 08:03:35PM 4 points [-]

Failing us relative to our expectations.

Comment author: Manfred 19 December 2011 08:20:31PM 2 points [-]

It's not particularly failing me relative to my expectations. And why does he use, say, the Pfizer executive's expectations as an example of something that science is failing by? "Our expectations" seems suspiciously similar to "all expectations ever." Or, more likely, "expectations the author thought it would be a good idea to have had of science when writing the article."

Comment author: FAWS 19 December 2011 08:35:02PM *  5 points [-]

Well, most people seem to be surprised that the majority of medical science results (or at least a high percentage) turns out to be bogus.

Comment author: Manfred 19 December 2011 11:26:09PM 1 point [-]

see: social institution vs. way of learning things.

Comment author: [deleted] 19 December 2011 09:20:13PM *  0 points [-]

the majority medical science results (or at least a high percentage) turns out to be bogus.

I assume that you really mean "the majority drug results (or at least a high percentage) turns to be ineffective"? A claim that is still far from uncontroversial.

Edit: Change "drug" results to "epidemiology".

Comment author: FAWS 19 December 2011 09:35:53PM *  1 point [-]

Drug results and correlation studies, both environmental and genetic, mostly. Which should be high enough volume that the "at least a high percentage" part should be true even if you add more reliable types of research, no? Or is medical science the wrong word for the category that includes both?

Comment author: [deleted] 19 December 2011 10:10:49PM 0 points [-]

How much is a high percentage?

Or is medical science the wrong word for the category that includes both?

I do think so. A lot of pre-clinical medical science is more about understanding specific mechanism, not looking at correlations and mapping out risk factors.

Drug results and correlation studies, both environmental and genetic, mostly,

Do you have some data? I do agree that it's hard to actually learn something solid from epidemiology, biology is complicated and factors do not usually add in any intuitive way. But then there are categories where epidemiology is invaluable take for example people with hereditary colon cancer where the majority (with a specific set of mutations) get colon cancer. But you might be right that a lot is not really useful information . . .

Comment author: gwern 19 December 2011 10:28:36PM 3 points [-]

Of course there is data. Besides the Ionnidis citations in the linked article, I also linked my previous post on the topic which, among other things, links to my section in the DNB FAQ on this topic with dozens of links/citations.

Comment author: [deleted] 19 December 2011 11:59:21PM 0 points [-]

My bad, only browsed through "Why Science Is Failing Us", behaved kind of like a politician, will do my homework before opening my mouth next time.

But I still think that one should use medical epidemiology instead of the cluster word medical science.

Comment author: FAWS 19 December 2011 10:27:48PM *  3 points [-]

How much is a high percentage?

Let's say more than 20%

I do think so. A lot of pre-clinical medical science is more about understanding specific mechanism, not looking at correlations and mapping out risk factors.

I didn't necessarily mean to exclude things like that, just to include both of the categories mentioned.

Do you have some data?

http://www.plosmedicine.org/article/info:doi/10.1371/journal.pmed.0020124

Comment author: Kaj_Sotala 20 December 2011 06:43:10PM 0 points [-]

How much is a high percentage?

From the article:

One study, for instance, analyzed 432 different claims of genetic links for various health risks that vary between men and women. Only one of these claims proved to be consistently replicable. Another meta review, meanwhile, looked at the 49 most-cited clinical research studies published between 1990 and 2003. Most of these were the culmination of years of careful work. Nevertheless, more than 40 percent of them were later shown to be either totally wrong or significantly incorrect.

Those didn't analyze all of medicine, of course, but it does sound pretty bad for the overall percentage.

Comment author: DanArmak 20 December 2011 12:07:56PM 3 points [-]

Then our expectations are wrong. The effectiveness of science should add up to normality.

Comment author: Eugine_Nier 21 December 2011 05:37:15AM 1 point [-]

Correct, of course that's still a problem.