= 27d567bc2d48d9f40e9ccc20ea370b15)
JoshuaZ comments on Should effective altruists care about the US gov't shutdown and can we do anything? - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
Comments (111)
Bad example. Einstein was A) doing physics at a time when the size of budgets needed to make new discoveries was much smaller B) primarily doing theoretical work or work that relied on other peoples data. Many areas of research (e.g. much of particle physics, a lot of condensed matter, most biology) require funding for the resources to simply to do anything at all.
I don't think that true.
If you take something like the highly useful discovery that taking vitamin D at the morning is more effective than at the evening that discovery was made in the last decade by amateurs without budjets.
Fermi estimates aren't easy but that discovery might be worth a year of lifespan. If you look at what the Google people are saying solving cancer is worth three years of lifespan. The people who publish breakthrough results in cancer research have replication rates of under 10 percent. Just as Petrov didn't get a nobel peace price, the people advancing human health don't get biology nobel prices.
Relying on other people's data is much easier know that it was in Einsteins time. Open science doesn't go as far as I would like but being able to transfer data easily via computers makes things so much easier.
The fact that most work in biology relies on experiments suggests that there are not enough people doing good theoretical work in the field it. I don't know much about particle phyiscs but I'm not sure whether we need as much smart people doing particle physics as we have at the moment.
So there are two distinct arguments being made: one is a resource allocation argument (it would be better to spend fewer resources right now on things like particle physics) and the second argument is that in many fields one can still make discoveries with few resources. The first argument may have some validity. The second argument ignores how much work is required in most cases. Yes, one can do things like investigate specific vitamin metabolism issues. But if one is interested in say synthesizing new drugs, or investigating how those drugs would actually impact people that requires large scale experiments.
That's not what is going on here. The issue is that biology is complicated. Life doesn't have easy systems that have easy theoretical underpinnings that can be easily computed. There are literally thousands of distinct chemicals in a cell interacting, and when you introduce a new one, even if you've designed it to interact with a specific receptor, it will often impact others. And even if it does only impact the receptor in question, how it does so will matter. You are dealing with systems created by the blind-idiot god.
You are defending a way of doing biology that plagued by various problems. It's a field where people literally believe that they can perceive more when they blind themselves.
There are huge issues in the theoretically underpinning of that approach because the people in the system are too busy writing research that doesn't replicate for top tier journals that requires expensive equipment instead of thinking more about how to approach the field.
So every field has problems, but that doesn't mean those problems are "huge".
Outside view: An entire field which is generally pretty successful at actually finding what is going on is fundamentally misguided about how they should be approaching the field, or the biologists are doing what they can. Biology is hard. But we are making progress in biology at a rapid rate. For example, the use of genetic markers to figure out how to treat different cancers was first proposed in the early 1990s and is now a highly successful clinical method.
Really? Can you point to a paper demonstrating it's better than classifying cancers the way histologists did in the 80s? Everything I've seen says that it just reconstructs the same classification. But it took ten years for the geneticists to admit that. I've seen more recent genetic classification that might be better than the old ones, but they didn't bother to compare to the old genetic classifications, let alone the histology.
I don't know enough about that subfield to answer that question. If what you are saying is accurate, that's highly disturbing. Most of my exposure to that subfield has been to popular press articles such as this one which paint a picture that sounds much more positive, but may well be highly distorted from what's actually going on.
HER2 receptor. These days those with breast cancer that overexpresses this growth factor receptor tend to get monoclonal antibodies against it, which both suppress its growth effects and tag it for disruption by the immune system.
Yes, this is a protein test rather than a genetic test. But it lets the subset of people with this amplification get a treatment that has a large positive absolute effect on those with early-stage cancer.
You might be but I'm not really.
That's a crude method of measuring success.
The cost of new drugs rises exponentially via Eroom's law. Big Pharma constantly lays of people.
A problem like obesity grows worse over the years instead of progress. Diabetes gets worse.
Even if you say that science isn't about solving real world issues but about knowledge, I also think that replication rates of 11% in the case of breakthrough cancer research indicates that the field is not good at finding out what's going on.
It isn't a metric of success. It is an example, one of many in the biological sciences.
This is likely due largely to policy issues and legal issues more than it is how the biologists are thinking. Clinical trials have gotten large.
A systemic problem, but one that has even less to do with biological research than Eroom's law. Obesity is not due to a lack of theoretical underpinnings in biology.
The question isn't is the field very good. The question is are the problems which we both agree exist due at all to not enough theory? File drawer effects, cognitive biases, bad experimental design are all issues here, none of which fall into that category.
Then at what grounds do you claim that the field is succesful? How would you know if it weren't succesful?
I'm not saying that theory lacks theoretical underpinnings but that the underpinning is of bad quality.
Question about designing experiments in a way that they produce reproduceable results instead of only large p values are theoretical issues.
Enough theory sounds like as attempt to quantify the amount of theory. That's not what I advocate. Theories don't get better through increase in their quantity. Good theoretical thinking can simply model and result in less complex theory.
That's a good question, but in this context, seeing a variety of novel discoveries in the last few years indicates a somewhat successful field. By the same token, I'm curious what makes you think this isn't a successful field?
I've already mentioned the file drawer problem. I'm curious, do you think that is a theoretical problem? If so, this may come down in part due to a very different notion of what theory means.
You seem to be treating biology to some extent like it is physics, But these are complex systems. What makes you think that such approaches will be at all successful?
The fact that Big Pharma has to lay of a lot of scientists is a real world indication that the output of model of finding a drug target, screening thousands of components against it, runs those components through clinical trials to find whether they are any good and then coming out with drugs that cure important illnesses at the other end stops producing results. Eroom's law.
Saying that there's a file drawer problem is quite easy. That however not a solution. I think your problem is that you can't imaging a theory that would solve the problem. That's typical. If it would be easy to imagine a theoretical breakthrough beforehand it wouldn't be much of a breakthrough.
Look at a theoretical breakthrough of moving from the model of numbers as IV+II=VI to 4+2=6. If you would have talked with a Pythagoras he probably couldn't imaging a theoretical breakthrough like that.
I don't. I don't know much about physics. Paleo/Quantified Self people found the thing with Vitamin D in the morning through phenemology. The community is relatively small and the amount of work that's invested into the theoretical underpinning is small.
I think in my exposure with the field of biology from various angles that there are a lot of areas where things aren't clear and there room for improvement on the level on epistomolgy and ontology.
I just recently preordered two angel sensors from crowdsourcing website indiegogo. I think that the money that the company gets will do much more to advance medicine than the average NHI grant.
No, seeing a bunch of novel true discoveries indicates a successful field. However, it's normally hard to independently verify the truth of novel discoveries except in cases where those discoveries have applications.
I don't think a flat replication rate of 11% tells us anything without recourse to additional considerations. It's sort of like a Umeshism: if your experiments are not routinely failing, you aren't really experimenting. The best we can say is that 0% and 100% are both suboptimal...
For example, if I was told that anti-aging research was having a 11% replication rate for its 'stopping aging' treatments, I would regard this as shockingly too high and a collective crime on par with the Nazis, and if anyone asked me, would tell them that we need to spend far far more on anti-aging research because we clearly are not trying nearly enough crazy ideas. And if someone told me the clinical trials for curing balding were replicating at 89%, I would be a little uneasy and wonder what side-effects we were exposing all these people to.
(Heck, you can't even tell much about the quality of the research from just a flat replication rate. If the prior odds are 1 in 10,000, then 11% looks pretty damn good. If the prior odds are 1 in 5, pretty damn bad.)
What I would accept as a useful invocation of an 11% rate is, say, an economic analysis of the benefits showing that this represents over-investment (for example, falling pharmacorp share prices) or surprise by planners/scientists/CEOs/bureaucrats where they had held more optimistic assumptions (and so investment is likely being wasted). That sort of thing.
Replication rate of experiments is quite different from the success rate of experiments.
An 11% success rate is often shockingly high. An 11% replication rate means the researchers are sloppy, value publishing over confidence in the results, and likely do way too much of throwing spaghetti at the wall...
Even granting your distinction, the exact same argument still applies: just substitute in an additional rate of, say, 10% chance of going from replication to whatever you choose to define as 'success'. You cannot say that a 11% replication rate and then a 1.1% success rate is optimal - or suboptimal - without doing more intellectual work!
No, I don't think so. An 11% replication rate means that 89% of the published results are junk and external observers have no problems seeing that. Which implies that if those who published it were a bit more honest/critical/responsible, they should have been able to do a better job of controlling for the effects which lead them to think there's statistical significance when in fact there's none.
If the prior odds are 1:10,000 you have no business publishing results at 0.05 confidence level.