LESSWRONG
LW

Jonas Hallgren — LessWrong

That is a fair point, since virtue is tied to your identity and self it is a lot easier to take things personally and therefore distort the truth.

A part of me is like "meh, skill issue, just get good at emotional intelligence and see through your self" but that is probably not a very valid solution at scale if I'm being honest.

There's still something nice about it leading to repeated games and similar, something about how if you look at our past then cooperation arises from repeated games rather than individual games where you analyse things in detail. This is the specific point that Joshua Greene makes in his Moral Tribes book... (read more)

Jonas Hallgren2d

I find this an interesting line of criticism, it is essentially pointing at the difficulty of finding good evidence and evaluating yourself on that evidence and making a disagreement about how easy it is.

I would like to bring in a perspective of more first principles modelling of how quickly you incorporate evidence pointing against the way you're thinking.

One thing is the amount of disconfirming evidence you look for. Another thing is your ability to bring that information into your worldview, your openness to being wrong. Thirdly we might also mention is the speed of the feedback, how long time does it take for you to get feedback.

I think you're saying that when... (read more)

Replying toSpectral Signatures of Gradual Disempowerment

Jonas Hallgren3d

Spectral Signatures of Gradual Disempowerment

Perhaps a multi-agent alignment research direction could be to create networks with higher cognitive complexity, with the goal of limiting the persuasion effect of any single agent? This is probably more compelling if you have a high probability of models being mostly aligned, and are misaligned in mostly distinct ways.

Yes! Very good! You spotted it too, a lot of the research I've been doing into this space is about how to build institutional structures that are resillient to capture of resources and behaviour. I've been working on setting up environments for a sort of misinformation evaluation of different types of institutions and social networks.

I'm basically in the camp of LLMs being semi-aligned... (read more)

Spectral Signatures of Gradual Disempowerment

Jonas Hallgren

TL;DR

AI disempowerment operates across markets, networks, and governance simultaneously, but our analytical tools don't cross those boundaries. We propose spectral graph metrics—spectral gap, Fiedler vector, eigenvalue distribution—as computable, cross-domain measures for tracking how the balance of influence shifts when AI enters coordination systems, and identify three specific quantities to monitor for AI governance.

Introduction

AI systems are changing how society coordinates — across markets, networks, governance institutions, scientific communities, all at once. The gradual disempowerment thesis captures why this is hard to address: human influence over collective outcomes can erode slowly, through ordinary competitive dynamics, without any single dramatic failure. AI systems become better at navigating coordination mechanisms, and the effective weight of human... (read 4954 more words →)

Replying toOn Goal-Models

Jonas Hallgren8d

On Goal-Models

I agree with you but I think you might be missing the point, let me add some detail from the cultural evolution angle. (because I do think the paper is actually implicitly saying something interesting about mean-field and similar approaches.)

One of the main points from Henrich's WEIRDest People in the World is between internalized identity and relational identity. WEIRD populations develop internal guilt, stable principles, context-independent rules—your identity lives inside you. Non-WEIRD populations have identities that exist primarily in relation to specific others: your tribe, your lineage, your patron. Who you are depends on who you're interacting with.

Now connect this to the effective horizon result. Consider the king example: if you send... (read more)

Replying toOn Goal-Models

Jonas Hallgren9d

On Goal-Models

It turns out that RL largely works in cases where "assume my future self plays optimally" is equivalent to "assume my future self plays randomly" (!)

This paper is really cool!

A random fun thing that I connected it to is how civilization can be seen as an engineering process that enables lower context horizon planning. The impersonality of a market is essentially a standardising force that allows you to not have to deal with long chains of interpersonal connections which enables you to model the optimal policy as close to a random policy for the average person will behave in a standardised way (to work for a profit).

This might connect to Henrich's stuff on market integration producing more impersonal/rule-based cognition. Maybe populations adapt to environments where random-policy-Q-functions are informative, and we call that adaptation "WEIRD psychology"?

Something something normalization and standardization as horizon-reduction...

Replying toThe Atoms of Knowledge Aren't Universal

Jonas Hallgren9d

The Atoms of Knowledge Aren't Universal

What comes to mind is Schrödinger's "What is Life?" (1944)—hugely influential for molecular biology, but Max Perutz wrote that "what was true in his book was not original, and most of what was original was known not to be true." Pauling called his thermodynamics "vague and superficial." (detailed review)

I think this could have been perceived and done better if Schrödinger spent some more time engaging with the axiomatic setup of biology.

Econophysics shows a similar pattern—physicists applying statistical mechanics to markets, largely ignored by economists who see them as missing institutional details. (Science coverage)

I asked Claude for further examples:

Sociobiology wars (1970s-80s): Wilson had water poured on him at a conference; accused of scientific racism by Harvard colleagues
Physicists entering neuroscience/consciousness research more broadly: pattern of proposing mechanisms without engaging with existing empirical constraints

The Atoms of Knowledge Aren't Universal

Jonas Hallgren

Why DeSci should stop searching for universal verification and start building compositional translations.

Introduction

Here's a problem nobody talks about enough in decentralized science: how do you get a biologist and a physicist to collaborate without someone eventually muttering "well, actually, it's all just atoms" and the other person leaving the room?

This isn't a joke (well, it kind of is but whatever). The history of interdisciplinary science is littered with promising collaborations that collapsed because one field's way of verifying truth felt like an insult to another's. The physicist thinks the biologist is being sloppy. The biologist thinks the physicist is missing the point. Both are, in a sense, correct—they're just operating at different... (read 3621 more words →)

Jonas Hallgren14d

I would see it more as casual learning with the do operator and so it isn't necessarily about fitting to a MSE but rather doing testing of different combinations?

Something something gears level models

Replying toDialogue: Is there a Natural Abstraction of Good?

Jonas Hallgren15d

Dialogue: Is there a Natural Abstraction of Good?

To summarise it, grow your honesty and ability to express your own wants instead of becoming more paranoid about whether you're being manipulated because it will probably be too late to do anyway?

That does make sense and seems like a good strategy.

Replying toDialogue: Is there a Natural Abstraction of Good?

Jonas Hallgren16d

Dialogue: Is there a Natural Abstraction of Good?

I think your points are completely valid, I funnily enough found myself doing a bunch of reasoning around something like "that's cool and all but I would know if I'm cognitively impaired when it comes to this", like llms aren't strong enough yet and similar but that's also something someone who gets pwned would say.

So basically it is a skill issue but it is a really hard skill and on priors most people don't have it. And for myself i shall now beware any long term planning based on emotional responses from myself about LLMs because I definetely have over 1000 hours with LLMs.

Replying toDialogue: Is there a Natural Abstraction of Good?

Jonas Hallgren16d

Dialogue: Is there a Natural Abstraction of Good?

This was quite fun to read and to take part of how it feels like to do 4d chess thinking in a way that I do not believe I'm capable of myself but I wanted to give a reflection anyway. (I do not think of my own epistemology in this sort of recursive way and so it is a bit hard to relate/understand fully but still very interesting.)

This is not about ideology. I have met many people who tell me "I would in fact change my job based on evidence."

I guess I'm resonating more with Davidad here and so this is basically a longer question to Gabriel.

I would want to give a... (read 560 more words →)

Crystals in NNs: Technical Companion Piece

Jonas Hallgren

2mo

This is the technical companion piece for Have You Tried Thinking About It As Crystals.

Epistemic Status: This is me writing out the more technical connections and trying to mathematize the undelying dynamics to make it actually useful. I've spent a bunch of time on Spectral Graph Theory & GDL over the last year so I'm confident in that part but uncertain in the rest. From the perspective of my Simulator Worlds framing this post is Exploratory (e.g I'm uncertain whether the claims are correct and it hasn't been externally verified) and it is based on an analytical world. Therefore, take it with a grain of salt and explore the claims as they... (read 4248 more words →)