I feel like this creates more misconceptions than it clears up. It's very dismissive of something that is really in the early phases of being studied.
The primary effect that reading this had on me was the change in state from [owning a cloak hadn't occurred to me] to [owning a cloak sounds awesome; i am unhappy that i hadn't thought of it on my own]
The definition of 'heritable' being underspecified (since you have to specify what population of environments you're considering) is not the same as being incoherent.
I agree. Good point.
So most of what you have written makes sense but there are some major issues with some parts.
heritability (a term that has an incoherent definition)
Can you expand on what you think about the definition is incoherent? This is a pretty standard term.
the whole gene vs. environment discussion is obsolete, in light of the findings of the past decade. Everything is gene-environment interaction.
The fact that many genes interact in a complicated way with the environment is not newly discovered. It doesn't change the fact that in some contexts genes or environment can matter more or less. For example, if one has a gene that codes for some form of mental retardation, in most cases, environment can't change that. (I say in most cases because there are a few exceptions especially related to issues related to trace nutrients or to bad reactions to specific compounds). Similarly, if someone has severe lead poisoning they are going to have pretty bad problems regardless of what the genes the person has.
The first two points you made while roughly valid connect to a more general issue- yes these studies have flaws, but just because a technique has flaws doesn't mean we can't use it to learn (especially when in this context the issues you bring up are well known to the researchers).
The answer to the question "what proportion of phenotypic variability is due to genetic variability?" always has the same answer: "it depends!" What population of environments are you doing this calculation over? A trait can go from close to 0% heritable to close to 100% heritable, depending on the range of environments in the sample. That's a definition problem. Further, what should we count as 'genetic'? Gene expression can depend on the environment of the parents, for example (DNA methylation, etc). That's an environmental inheritance. I just think there is an old way of talking about these things that needs to go away in light of current knowledge.
I agree with you that we still can learn a lot from these studies.
Adoption studies are biased toward the null of no parenting effect, because adoptive parents aren't randomly selected from the population of potential parents (they often are screened to be similar to biological parents).
Twin studies I think are particularly flawed when it comes to estimating heritability (a term that has an incoherent definition). Twins have a shared pre-natal environment. In some cases, they even share a placenta.
Plus, the whole gene vs. environment discussion is obsolete, in light of the findings of the past decade. Everything is gene-environment interaction.
wait, this isn't well done satire?
Informed consent bias in RCTs?
The problem of published research findings not being reliable has been discussed here before.
One problem with RCTs that has received little attention is that, due to informed consent laws and ethical considerations, subjects are aware that they might be receiving sham therapy. This differs from the environment outside of the research setting, where people are confident that whatever their doctor prescribes is what they will get from their pharmacist. I can imagine many ways in which subjects' uncertainty about treatment assignment could affect outcomes (adherence is one possible mechanism). I wrote a short paper about this, focusing out what we would ideally estimate if we could lie to subjects, versus what we actually can estimate in RCTs (link). Here is the abstract:
It is widely recognized that traditional randomized controlled trials (RCTs) have limited generalizability due to the numerous ways in which conditions of RCTs differ from those experienced each day by patients and physicians. As a result, there has been a recent push towards pragmatic trials that better mimic real-world conditions. One way in which RCTs differ from normal everyday experience is that all patients in the trial have uncertainty about what treatment they were assigned. Outside of the RCT setting, if a patient is prescribed a drug then there is no reason for them to wonder if it is a placebo. Uncertainty about treatment assignment could affect both treatment and placebo response. We use a potential outcomes approach to define relevant causal effects based on combinations of treatment assignment and belief about treatment assignment. We show that traditional RCTs are designed to estimate a quantity that is typically not of primary interest. We propose a new study design that has the potential to provide information about a wider range of interesting causal effects
Any thoughts on this? Is this a trivial technical issue or something worth addressing?
Error detection bias in research
I have had the following situation happen several times during my research career: I write code to analyze data; there is some expectation about what the results will be; after running the program, the results are not what was expected; I go back and carefully check the code to make sure there are no errors; sometimes I find an error
No matter how careful you are when it comes to writing computer code, I think you are more likely to find a mistake if you think there is one. Unexpected results lead one to suspect a coding error more than expected results do.
In general, researchers usually do have general expectations about what they will find (e.g., the drug will not increase risk of the disease; the toxin will not decrease risk of cancer).
Consider the following graphic:
Here, the green region is consistent with what our expectations are. For example, if we expect a relative risk (RR) of about 1.5, we might not be too surprised if the estimated RR is between (e.g.) 0.9 and 2.0. Anything above 2.0 or below 0.9 might make us highly suspicious of an error -- that's the red region. Estimates in the red region are likely to trigger serious coding error investigation. Obviously, if there is no coding error then the paper will get submitted with the surprising results.
Bayes' rule =/= Bayesian inference
Related to: Bayes' Theorem Illustrated, What is Bayesianism?, An Intuitive Explanation of Bayes' Theorem
(Bayes' theorem is something Bayesians need to use more often than Frequentists do, but Bayes' theorem itself isn't Bayesian. This post is meant to be a light introduction to the difference between Bayes' theorem and Bayesian data analysis.)
Bayes' Theorem
Bayes' theorem is just a way to get (e.g.) p(B|A) from p(A|B) and p(B). The classic example of Bayes' theorem is diagnostic testing. Suppose someone either has the disease (D+) or does not have the disease (D-) and either tests positive (T+) or tests negative (T-). If we knew the sensitivity P(T+|D+), specificity P(T-|D-) and disease prevalence P(D+), then we could get the positive predictive value P(D+|T+) using Bayes' theorem:
For example, suppose we know the sensitivity=0.9, specificity=0.8 and disease prevalence is 0.01. Then,
This answer is not Bayesian or frequentist; it's just correct.
Diagnostic testing study
Typically we will not know P(T+|D+) or P(T-|D-). We would consider these unknown parameters. Let's denote them by Θsens and Θspec. For simplicity, let's assume we know the disease prevalence P(D+) (we often have a lot of data on this).
Suppose 1000 subjects with the disease were tested, and 900 of them tested positive. Suppose 1000 disease-free subjects were tested and 200 of them tested positive. Finally, suppose 1% of the population has the disease.
Frequentist approach
Estimate the 2 parameters (sensitivity and specificity) using their sample values (sample proportions) and plug them in to Bayes' formula above. This results in a point estimate for P(D+|T+) of 0.043. A standard error or confidence interval could be obtained using the delta method or bootstrapping.
Even though Bayes' theorem was used, this is not a Bayesian approach.
Bayesian approach
The Bayesian approach is to specify prior distributions for all unknowns. For example, we might specify independent uniform(0,1) priors for Θsens and Θspec. However, we should expect the test to do at least as good as guessing (guessing would mean randomly selecting 1% of people and calling them T+). In addition, we expect Θsens>1-Θspec. So, I might go with a Beta(4,2.5) distribution for Θsens and Beta(2.5,4) for Θspec:
Using these priors + the data yields a posterior distribution for P(D+|T+) with posterior median 0.043 and 95% credible interval (0.038, 0.049). In this case, the Bayesian and frequentist approaches have the same results (not surprising since the priors are relatively flat and there are a lot of data). However, the methodology is quite different.
Example that illustrates benefit of Bayesian data analysis
(example edited to focus on credible/confidence intervals)
Suppose someone shows you what looks like a fair coin (you confirm head on one side tails on the other) and makes the claim: "This coin will land with heads up 90% of the time"
Suppose the coin is flipped 5 times and lands with heads up 4 times.
Frequentist approach
"A 95% confidence interval for the Binomial parameter is (.38, .99) using the Agresti-Coull method." Because 0.9 is within the confidence limits, the usual conclusion would be that we do not have enough evidence to rule it out.
Bayesian approach
"I don't believe you. Based on experience and what I know about the laws of physics, I think it's very unlikely that your claim is accurate. I feel very confident that the probability is close to 0.5. However, I don't want to rule out something a little bit unusual (like a probability of 0.4). Thus, my prior for the probability of heads is a Beta(30,30) distribution."
After seeing the data, we update our belief about the binomial parameter. The 95% credible interval for it is (0.40, 0.64). Thus, a value of 0.9 is still considered extremely unlikely.
This illustrates the idea that, from a Bayesian perspective, implausible claims require more evidence than plausible claims. Frequentists have no formal way of including that type of prior information.

View more: Next






Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)
If you are not going to do an actual data analysis, then I don't think there is much point of thinking about Bayes' rule. You could just reason as follows: "here are my prior beliefs. ooh, here is some new information. i will now adjust my believes, by trying to weigh the old and new data based on how reliable and generalizable i think the information is." If you want to call epistemology that involves attaching probabilities to beliefs, and updating those probabilities when new information is available, 'bayesian' that's fine. But, unless you have actual data, you are just subjectively weighing evidence as best you can (and not really using Bayes' rule).
The thing that can be a irritating is when people then act as if that kind of reasoning is what bayesian statisticians do, and not what frequentist statisticians do. In reality, both types of statisticians use Bayes' rule when it's appropriate. I don't think you will find any statisticians who do not consider themselves 'bayesian' who disagree with the law of total probability.
If you are actually going to analyze data and use bayesian methods, you would end up with a posterior distribution (not simply a single probability). If you simply report the probability of a belief (and not the entire posterior distribution), you're not really doing conventional bayesian analysis. So, in general, I find the conventional Less Wrong use of 'bayesian' a little odd.