aarongertler

One Study, Many Results (Matt Clancy)

I didn't see this post, its author, or the study involved elsewhere on LW, so I'm crossposting the content. Let me know if this is redundant, and I'll take it down. Summary This post looks at cases where teams of researchers all began with the same data, then used it to answer a question — and got a bunch of different answers, based on their different approaches to statistical testing, "judgment calls", etc. This shows the difficulty of doing good replication work even without publication bias; none of the teams here had any special incentive to come up with a certain result, and they all seemed to be doing their best to really answer the question. Also, I'll copy the conclusion of the post and put it here: > More broadly, I take away three things from this literature: > > 1. Failures to replicate are to be expected, given the state of our methodological technology, even in the best circumstances, even if there’s no publication bias. > 2. Form your ideas based on suites of papers, or entire literatures, not primarily on individual studies. > 3. There is plenty of randomness in the research process for publication bias to exploit. More on that in the future. The post Science is commonly understood as being a lot more certain than it is. In popular science books and articles, an extremely common approach is to pair a deep dive into one study with an illustrative anecdote. The implication is that’s enough: the study discovered something deep, and the anecdote made the discovery accessible. Or take the coverage of science in the popular press (and even the academic press): most coverage of science revolves around highlighting the results of a single new (cool) study. Again, the implication is that one study is enough to know something new. This isn’t universal, and I think coverage has become more cautious and nuanced in some outlets during the era of covid-19, but it’s common enough that for many people “believe science” is a sincere mantra, as if science m

84Jul 18, 2021

aarongertler

Message

I work for Open Philanthropy. I spent 2019-2021 at the Centre for Effective Altruism, where I ran the EA Forum. I'm also a semi-professional gamer and streamer who placed second at the 2020 Magic: the Gathering world championship ("Grand Finals").

648

Open Philanthropy is hiring for multiple roles across our Global Catastrophic Risks teams

It’s been another busy year at Open Philanthropy; after nearly doubling the size of our team in 2022, we’ve added over 30 new team members so far in 2023. Now we’re launching a number of open applications for roles in all of our Global Catastrophic Risks (GCR) cause area teams...

Oct 4, 20236

EA Forum Creative Writing Contest: $10,000 in prizes for good stories

We just launched a creative writing contest on the Effective Altruism Forum. Stories like HPMOR and The Fable of the Dragon-Tyrant have been massively impactful, and we'd like to see more work in that vein — please consider submitting something! Note that you can also submit past work that seems...

Sep 12, 202132

One Study, Many Results (Matt Clancy)

Jul 18, 202184

Tom Chivers, author of "The AI Does Not Hate You", is running an AMA on the EA Forum

Tom's book on the rationalist community was generally well-received by community members. I enjoyed the book, and have continued to enjoy following Tom's work on COVID and many other things. Among science journalists, I find him to be an unusually clear writer, with good taste when it comes to choosing...

Mar 11, 202114

aarongertler's Shortform

Jan 6, 20214

What are good ways of convincing someone to rethink an impossible dream?

I sometimes get pitches from people who have grandiose plans. Think of the term "physics crank," but applied to conceptual companies rather than perpetual motion machines. Two prominent examples (but far from the only ones): * A calligrapher and accountant with no entrepreneurial experience who wanted to create an electronic...

Mar 19, 202014

Who are your favorite "hidden rationalists"?

Quick summary: "Hidden rationalists" are what I call authors who espouse rationalist principles, and probably think of themselves as rational people, but don't always write on "traditional" Less Wrong-ish topics and probably haven't heard of Less Wrong. I've noticed that a lot of my rationalist friends seem to read the...

Jan 11, 201525

Load More (7/12)

LESSWRONG
LW

LESSWRONG
LW

aarongertler

aarongertler

aarongertler

One Study, Many Results (Matt Clancy)

EA Forum Creative Writing Contest: $10,000 in prizes for good stories

Who are your favorite "hidden rationalists"?

Buying Debt as Effective Altruism?

aarongertler

Open Philanthropy is hiring for multiple roles across our Global Catastrophic Risks teams

EA Forum Creative Writing Contest: $10,000 in prizes for good stories

One Study, Many Results (Matt Clancy)

Tom Chivers, author of "The AI Does Not Hate You", is running an AMA on the EA Forum

aarongertler's Shortform

What are good ways of convincing someone to rethink an impossible dream?

Who are your favorite "hidden rationalists"?

One Study, Many Results (Matt Clancy)

EA Forum Creative Writing Contest: $10,000 in prizes for good stories

Who are your favorite "hidden rationalists"?

Buying Debt as Effective Altruism?

Open Philanthropy is hiring for multiple roles across our Global Catastrophic Risks teams

EA Forum Creative Writing Contest: $10,000 in prizes for good stories

One Study, Many Results (Matt Clancy)

Tom Chivers, author of "The AI Does Not Hate You", is running an AMA on the EA Forum

aarongertler's Shortform

What are good ways of convincing someone to rethink an impossible dream?

Who are your favorite "hidden rationalists"?