LESSWRONG
LW

Siebe — LessWrong

Oh, oops. I'm happy to do that, although the post will not "make sense without expanding the AI-text section"

An epistemic theory of populism [link post to Joseph Heath]

4mo

The following is an AI-generated summary

Populism Fast and Slow - Summary

Joseph Heath argues that existing academic definitions of populism—whether treating it as an ideology or merely an electoral strategy—are inadequate. He proposes understanding populism through the lens of dual-process cognitive theory (Kahneman's "fast and slow" thinking).

Core Argument

Populism is a political strategy that privileges intuitive cognition (System 1: fast, effortless, "common sense") over analytical reasoning (System 2: slow, effortful, requiring expertise). This creates a fundamental divide between "the people" who rely on intuition and "elites" who employ analytical thinking.

Why This Matters

Many evolved cognitive intuitions work poorly in modern large-scale societies. For example:

People intuitively overestimate punishment's effectiveness (due to regression-to-mean bias)
They misunderstand trade and immigration economics
They struggle with collective action problems

When experts develop... (read more)

Siebe6moQuick Take

Substack continues growing, it looks sustainable and profitable, and unlike BlueSky isn't attracting only one part of the political spectrum. I don't see Substack doing much to raise the sanity waterline, but it might be a worthy target for some rationalist lobbying to improve their architecture.

Siebe7moQuick Take

Shallow take:

I feel iffy about negative reinforcement still being widely used in AI. Both human behaviour experts (child-rearing) and animal behavior experts seem to have largely moved away from that being effective, only leading to unwanted behavior down the line

Siebe8moQuick Take

There's a number of priors that lead me to expect much of the current AI safety research to be low quality:

A lot of science is low quality. It's the default expectation for a research field.
It's pre-paradigmatic. Norms haven't been established yet for what works in the real world, what are reliable methods and what is p-hacking etc. This makes it not only difficult to produce good work, it also makes it hard to recognize bad work and hard to get properly calibrated about how much work is bad, the way we are in established research fields.
It's subject to selection effects by non-experts. It gets amplified by advocates, journalists, policy groups, the general

Siebe10mo

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

More thoughts:

I thought that AlphaZero was a counterpoint, but apparently it's significantly different. For example, it used true self-play allowing it to discover fully novel strategies.

Then again, I don't think more sophisticated reasoning is the bottleneck to AGI (compared to executive function & tool use), so even if reasoning doesn't really improve for a few years we could get AGI.

However, I previously thought reasoning models could be leveraged to figure out how to achieve actions, and then the best actions would be distilled into a better agent model, you know, IDA-style. But this paper makes me more skeptical of that working, because these agentic steps might require novel skills that aren't inside the training data.

Replying toDoes Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

Siebe10mo*

Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?

Yes it matters for current model performance, but it means that RLVR isn't actually improving the model in a way that can be used for an iterated distillation & amplification loop, because it doesn't actually do real amplification. If this turns out right, it's quite bearish for AI timelines

Edit: Ah someone just alerted me to the crucial consideration that this was tested using smaller models (like Qwen-2.5 (7B/14B/32B) and LLaMA-3.1-8B, which are significantly smaller than the models where RLVR has shown the most dramatic improvements (like DeepSeek-V3 → R1 or GPT-4o → o1). And given that different researchers have claimed that there's a threshold effect, substantially weakens these findings. But they say they're currently evaluating DeepSeek V3- & R1 so I guess we'll see

Replying toLearned pain as a leading cause of chronic pain

Siebe10mo

Learned pain as a leading cause of chronic pain

That's good to know.

For what it's worth, ME/CFS (a disease/cluster of specific symptoms) is quite different from idiopathic chronic fatigue (a single symptom). Confusing the two is one of the major issues in the literature. Many people with ME/CFS, like I, don't even have 'feeling tired' as a symptom. Which is why I avoid the term CFS.

Replying toLearned pain as a leading cause of chronic pain

Siebe10mo

Learned pain as a leading cause of chronic pain

I haven't looked into this literature, but it sounds remarkably similar to the literature of cognitive behavioral therapy and graded exercise therapy for ME/CFS (also sometimes referred to as 'chronic fatigue syndrome'). I can imagine this being different for pain which could be under more direct neurological control.

Pretty much universally, this research was of low to very low quality. For example, using overly broad inclusion criteria such that many patients did not have the core symptom of ME/CFS, and only reporting subjective scores (which tend to improve) while not reporting objective scores. These treatments are also pretty much impossible to blind. Non-blinding + subjective self-report is a pretty bad combination. This, plus... (read more)

I'm starting a discussion group on Signal to explore and understand the democratic backsliding of the US at ‘gears-level’. We will avoid simply discussing the latest outrageous thing in the news, unless that news is relevant to democratic backsliding.

Example questions:

“how far will SCOTUS support Trump's executive overreach?”
“what happens if Trump commands the military to support electoral fraud?”
"how does this interact with potentially short AGI timelines?”
"what would an authoritarian successor to Trump look like?"
"are there any neglected, tractable, and important interventions?"

You can join the group here: https://signal.group/#CjQKIE2jBWwjbFip5-kBnyZHqvDnxaJ2VaUYwbIpiE-Eym2hEhAy21lPlkhZ246_AH1V4-iA (If the link doesn't work anymore in the future, DM me.)

Matthew Yglesias - Misinformation Mostly Confuses Your Own Side

Siebe

Matthew Yglesias' post, Misinformation Mostly Confuses Your Own Side, argues that political misinformation tends to harm the group spreading it more than it persuades opponents. Key points:

Misinformation primarily affects in-group thinking: Supporters of a politician or movement are more likely to believe falsehoods from their own side, leading to internal confusion and strategic missteps.
Limited impact on opponents: While misinformation can provoke outrage, it rarely converts skeptics or shifts public opinion in a meaningful way.
Echo chambers reinforce bad strategy: When false beliefs circulate unchecked within a political movement, they can lead to poor decision-making and unrealistic expectations.
Honesty is both ethical and pragmatic: Yglesias argues that promoting accurate information strengthens a movement’s credibility and effectiveness over time.

(Read more)

Siebe1yQuick Take

This might be a stupid question, but has anyone considered just flooding LLM training data with large amounts of (first-person?) short stories of desirable ASI behavior?

The way I imagine this to work is basically that an AI agent would develop really strong intuitions that "that's just what ASIs do". It might prevent it from properly modelling other agents that aren't trained on this, but it's not obvious to me that that's going to happen or that it's such a decisively bad thing to outweigh the positives

Siebe's Shortform

Siebe

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Your risk of developing long COVID is probably high

Siebe

Context: I wrote the following on Facebook for a general audience. I'm posting it here because I'm very concerned. I'm severely disabled from a presumable* COVID infection 17 months ago. I haven't done deep dives into (m)any of these, so I'm plausibly wrong. However, this is the main position in the long COVID community. Sorry not sorry that it isn't polished. This post does not include new strains, which might also be worrisome (https://twitter.com/PeacockFlu/status/1504158873938272269?t=y2l5alk3ZjUOaDghjquUrg&s=08).

Summary:

Please keep protecting yourself (and others)! 🙏🏻😷
I estimate you're at 1-15% risk of developing long-term disability, at the population's average exposure rate
Successive organ and immune damage from infection is worrisome
Limit exposure, detect early and try to treat early

—--

Today, mask... (read 684 more words →)

A case for strategy research: what it is and why we need more of it

Siebe

Authors: Siebe Rozendal, Justin Shovelain, David Kristoffersson

Crossposted to the Effective Altruism Forum

Overview

To achieve any ambitious goal, some strategic analysis is necessary. Effective altruism has ambitious goals and focuses heavily on doing research. To understand how to best allocate our time and resources, we need to clarify what our options in research are. In this article, we describe strategy research and relate it to values research, tactics research, informing research, and improvement research. We then apply the lens of strategy research to existential risk reduction, a major cause area of effective altruism. We propose a model in which the marginal value of a research type depends strongly on the maturity of the research... (read 5901 more words →)

Welcome to Effective Altruism Groningen

Siebe

Hey!

We are EA Groningen, a small effective altruism community in the northern Dutch student town Groningen. We meet at least once a month in our regular pub: the Harbour Café. Besides that we organize lectures and learning meetups, and sometimes we have some projects running.

The people who regularly attend are between 20-40 years old, with about 60% students. Not many people read LW or SSC (besides me), but we welcome anyone who does and is interested in improving the world most effectively! Our cause interest is very wide: some interest in near term human welfare, some in animal welfare, some in long-term future, and many people who are unsure which is best :)

As long as you are open for new information and like to learn instead of only making your case, you are more than welcome. There is no background reading required, although reading some introductory EA material definitely enhances the conversations!

Check out our Meetup page for our upcoming events.