braces

P-hacking as focusing a microscope

Thanks, I agree that I should better understand why so many medical researchers do this. And I can definitely imagine situations where it helps: small sample (so ITT is under powered), useful treatment (so we’re worried about false negatives), low chance of bias (compliance isn’t correlated with outcomes). My belief is that these three factors don’t align enough to justify per protocol analysis. It’s way more likely to lead you to false cures. It’s hard to test this empirically though so if you had examples of treatments that got correctly assessed and advanced with a per protocol analysis I’d be very interested!

Where have all the tokens gone?

3mo

Science is full of misleading findings—results that would not hold up if the study were attempted again under ideal conditions. In data-driven investigations, a big part of this could be the forking paths problem. Researchers make many decisions about how to analyze the data (leave out this subgroup, include that control variable, etc.). But there are many defensible paths to take, and they could yield vastly different results. When the study gets published, you don’t learn about results from the paths not taken.

I think a lot of well-meaning scientists fall prey to this and churn out shoddy results. I don’t think they have an evil mindset, however. (One could certainly imagine a Psychopath's... (read 522 more words →)

Replying toWhere have all the tokens gone?

braces9mo

Thanks but doesn't pace of adoption break the analogy? Seems safe to say that those technologies caused growth, and so will AI. But AI has been rolled out fast enough that you might have seen near-term changes in the aggregate statistics, in contrast to physical or legal infrastructure. We do have careful evidence on railroads and the printing press though---is it wild to think that an information technology spreading some large multiple faster than the printing press would have impacts in the following year? Or is it that it's not such a technology shock at all? (Also, where do you think the education went?)

Replying toWhere have all the tokens gone?

braces9mo

Where have all the tokens gone?

Great point...are there certain consistent families of GPT-4-era hallucinations that you think would be especially important in this context?

Where have all the tokens gone?

Causal inference for the home gardener

9mo

In 2001, Lant Pritchett asked, “Where has all the education gone?” From 1960 to 1990, countries around the world had achieved increases in educational attainment without the widely expected gains in income—a “micro macro paradox” where the person-level estimates of the return to education were nowhere to be seen in aggregate output.

It seems we are on the cusp of another paradox in developing countries. People have an oracle in their pocket and it hasn’t seemed to help.

Is this surprising? The core premises are:

Human capital—or intelligence more generally—causes economic growth
AI models can substitute for human capital and intelligence
AI models have seen widespread use
We have not witnessed large gains in growth in developing countries

Together, these cannot all... (read 1651 more words →)

Reading RFK Jr so that you don’t have to

Note: This is meant to be an accessible introduction to causal inference. Comments appreciated.

Let’s say you buy a basil plant and put it on the counter in your kitchen. Unfortunately, it dies in a week.

So the next week you buy another basil plant and feed it a special powder, Vitality Plus. This second plant lives. Does that mean Vitality Plus worked?

Not necessarily! Maybe the second week was a lot sunnier, you were better about watering, or you didn’t grab a few leaves for a pasta. In other words, it wasn’t a controlled experiment. If some other variable like sun, water, or pasta is driving the results you’re seeing, your study is confounded, and... (read 1484 more words →)

Some people in my orbit suggested reading Robert F. Kennedy Jr’s book The Real Anthony Fauci.

Here’s my story of wading through a few pages of the book and trying to understand the basis of the claims. My takeaway: there's a lot of sloppiness here, and several times the book’s claim isn’t supported by the citation.

Often, though, the citation is fine. The arguments are carried by shoddy science and massive leaps in causal attribution. For a lot of the anti-vaccine and anti-lockdown takes, there’s at least some study by some scientists showing really bad effects. And the scientists are real—it’s just a small sample or poor design.

Sorting things out honestly is hard, especially with... (read 2292 more words →)

Replying toPer protocol analysis as medical malpractice

braces2y

I don’t think this part is so bad. You can lack “real control” but still have an informative experiment. You just need that people in the treatment group were on average more likely to receive the treatment. The issue is how they analyzed the data.

Replying toPer protocol analysis as medical malpractice

braces2y

It's a good question. I have the intuition that just a little potential for bias can go a long way toward messing up the estimated effect, so allowing this practice is net negative despite the gains in power. The dropouts might be similar on demographics but not something unmeasured like motivation. My view comes from seeing many failed replications and playing with datasets when available, but I would love to be able to quantify this issue somehow...I would certainly predict that studies where the per protocol finding differs from the ITT will be far less likely to replicate.

Replying toPer protocol analysis as medical malpractice

braces2y*

I don't have a sense of the overall prevalence, I'm curious about that too! I've just seen it enough in high-profile medical studies to think it's still a big problem.

Yes this is totally related to two-stage least squares regression! The intent-to-treat estimate just gives you the effect of being assigned to treatment. The TSLS estimate scales up the intent-to-treat by the effect that the randomization had on treatment (so, e.g., if the randomization increased the share doing yoga from 10 in the control group to 50% in the treatment group, the intent-to-treat effect divided by 0.40 would give you the TSLS estimate).

“Per protocol analysis” is when medical trial researchers drop people who didn’t follow the treatment protocol. It is outrageous and must be stopped.

For example: let’s say you want to know the impact of daily jogs on happiness. You randomly instruct 80 people to either jog daily or to simply continue their regular routine. As a per protocol analyst, you drop the many treated people who did not go jogging. You keep the whole control group because it wasn’t as hard for them to follow instructions.

At this point, your experiment is ruined. You ended up with lopsided groups: people able to jog and the unfiltered control group. It would not be surprising if... (read 273 more words →)

Replying toHow to have Polygenically Screened Children

How to have Polygenically Screened Children

FWIW, here's a Cochrane review on RCTs of such encouragement designs. It's basically a failed meta-analysis in that the data is too spotty to make any conclusions. But it shows at the very least that such studies have been done before. An encouragement design in a population with high alcohol use seems most promising for figuring out the causal effect on fetuses, as you say.

(Another recent meta-analysis on the same question finds a slight decrease in preterm birth in the alcohol education group, but this is based on just three studies and marginally significant so I'd say it's still uncertain.)

Are men harder to help?

Why is fiber good for you?

Of Boys and Men is a new book about how the outcomes of American men and boys are lagging in several ways. Most of it concerns aggregate trends that are irrefutable, such as the gender gap in college graduation. But books like these are often useful for testable hypotheses rather than reliable facts. Case in point: Reeves makes the odd claim that "men are hard to help":

[A] startling number of social programs seem to work well for girls and women, but not for boys and men — among which are a student-mentoring scheme in Fort Worth, Texas; a school-choice program in Charlotte, North Carolina; an income boost to low-wage earners in New York City;

... (read 400 more words →)

Fiber is defined as the parts of plants that we can eat but not digest. We're always told to eat more of it. Why should we eat things we can't digest?

There are many papers to point to. Fiber consumption is strongly associated with less cancer, less heart disease, and less death (recent meta-analysis here). Most of these findings come from observational studies, where people report their diets to researchers. Studies in which fiber consumption is experimentally controlled tend to find fewer benefits, so people who eat more fiber may already be healthier for other reasons.

Sidestepping the causal question, we could still ask: what are the plausible mechanisms? How does fiber do this?

Below... (read 313 more words →)

Replying toHow Risky Is Trick-or-Treating?

How Risky Is Trick-or-Treating?

Within years, the combo of ST_CASE and STATE uniquely identifies accidents, so an intermediate step is tracking the accident ID. Then you could classify accidents as involving a child as those where at least one of the decedents is under a certain age, then use that PER_TYP variable to look at the ages of the driver(s) for those accidents. Cool analysis, btw--I've worked a lot with the FARS and happy to help :)

Replying toHow Risky Is Trick-or-Treating?

How Risky Is Trick-or-Treating?

You can get the age of the driver by looking at the age of the person with PER_TYP equal to one.

Replying toAre c-sections underrated?

Are c-sections underrated?

Super useful, I would love to read more of your IVF thoughts if they're posted somewhere! I think your views imply that these new sperm obstacle courses, based on fears of a worse sperm-selection process in IVF, have zero effect on people's eventual outcomes (conditional on viability)? But curious what you think. Example: "The SPARTAN system uses a series of obstacles on a microchip, requiring sperm to swim around pillar-shaped objects and through the device. As a result, this device promotes the collection of the highest quality sperm."

Are c-sections underrated?

The ethics of reclining airplane seats

Summary

I think so. Correlations tend to show worse outcomes for c-sections, but the guidelines take this evidence too literally. Careful causal evidence finds that it’s more of a mixed bag. Moderate negative effects on subsequent births seem more robust. An elective c-section might be the defensible choice for someone’s final pregnancy.

Disclaimer: Not a doctor.

The official guidelines

C-sections cost about twice as much as vaginal delivery and are associated with worse health outcomes for infants and moms. These facts are concerning enough to the US Medicaid system that one of their “Improvement Initiatives” is reducing low-risk cesarean delivery. They write: “Cesarean delivery poses a greater risk of maternal morbidity and mortality for low-risk pregnancies when compared to... (read 1659 more words →)

Costs and benefits of amniocentesis for normal pregnancies

I enjoyed reading the replies to this tweet, since it's a lower stakes issue that has all the contours of broader ethical debates. Granted, what triggered the tweet was not low stakes:

There are passionately held beliefs on both sides (see replies to the original tweet for more). As with any argument, different principles lead to different conclusions. One principle with many adherents is that the rules follow directly from the design of the airplane:

But this could still screw over the person behind you. So maybe reclining is bad, based on the fact that it harms more than it helps?:

These views, by the way, are similar to what a travel industry analyst says... (read 260 more words →)