Dalcy

An Illustrated Summary of "Robust Agents Learn Causal World Model"

This post was written during Alex Altair's agent foundations fellowship program, funded by LTFF. Thank you Alex Altair, Alfred Harwood, Daniel C for feedback and comments. Introduction The selection theorems agenda aims to prove statements of the following form: "agents selected under criteria X has property Y," where Y are things such as world models, general purpose search, modularity, etc. We're going to focus on world models. But what is the intuition that makes us expect to be able to prove such things in the first place? Why expect world models? Because: assuming the world is a Causal Bayesian Network with the agent's actions corresponding to the D (decision) node, if its actions can robustly control the U (utility) node despite various "perturbations" in the world, then intuitively it must have learned the causal structure of how U's parents influence U in order to take them into account in its actions. And the same for the causal structure of how U's parents' parents influence U's parents ... and by induction, it must have further learned the causal structure of the entire world upstream of the utility variable. This is the intuitive argument that the paper Robust Agents Learn Causal World Model by Jonathan Richens and Tom Everitt formalizes. Informally, its main theorem can be translated as: if an agent responds to various environment interventions by prescribing policies that overall yield low regret, then it's possible to appropriately query the agent to reconstruct an implicit world model that matches up with the ground truth causal structure. I will refer to this result as the "Causal Good Regulator Theorem". This sequence Thoughts on the Causal Good Regulator Theorem will do the following: * [1] a precise statement of the theorem alongside a high-level argument of its proof and discussions of the paper's results in the context of the selection theorem agenda - basically a self-complete distillation of the paper. The rest of the posts in this

73Dec 14, 2024

What's the Right Way to think about Information Theoretic quantities in Neural Networks?

45Jan 19, 2025

When Are Results from Computational Complexity Not Too Coarse?

42Jul 3, 2024

But Where do the Variables of my Causal Model come from?

40Aug 9, 2024

Dalcy

Message

"I always remember, [Hamming] would come into my office and try to solve a problem [...] I had a very big blackboard, and he’d start on one side, write down some integral, say, ‘I ain’t afraid of nothin’, and start working on it. So, now,

...

1103

154

Towards building blocks of ontologies

This dialogue is part of the agent foundations fellowship with Alex Altair, funded by the LTFF. Thank you Dalcy, Alex Altair and Alfred Harwood for feedback and comments. Context: I (Daniel) am working on a project about ontology identification. I've found conversations to be a good way to discover inferential...

Feb 8, 202529

What's the Right Way to think about Information Theoretic quantities in Neural Networks?

Tl;dr, Neural networks are deterministic and sometimes even reversible, which causes Shannon information measures to degenerate. But information theory seems useful. How can we square this (if it's possible at all)? The attempts so far in the literature are unsatisfying. Here is a conceptual question: what is the Right Way...

Jan 19, 202545

Proof Explained for "Robust Agents Learn Causal World Model"

This post was written during Alex Altair's agent foundations fellowship program, funded by LTFF. Thank you Alex Altair, Alfred Harwood, Daniel C for feedback and comments. This is a post explaining the proof of the paper Robust Agents Learn Causal World Model in detail. Check the previous post in the...

Dec 22, 202428

An Illustrated Summary of "Robust Agents Learn Causal World Model"

Dec 14, 202473

Money Pump Arguments assume Memoryless Agents. Isn't this Unrealistic?

I have been reading about money pump arguments for justifying the VNM axioms, and I'm already stuck at the part where Gustafsson justifies acyclicity. Namely, he seems to assume that agents have no memory. Why does this make sense?[1] To elaborate: The standard money pump argument looks like this. >...

Aug 16, 202429

But Where do the Variables of my Causal Model come from?

Tl;dr, A choice of variable in causal modeling is good if its causal effect is consistent across all the different ways of implementing it in terms of the low-level model. This notion can be made precise into a relation among causal models, giving us conditions as to when we can...

Aug 9, 202440

Why do Minimal Bayes Nets often correspond to Causal Models of Reality?

Chapter 2 of Pearl's Causality book claims you can recover causal models given only the observational data, under very natural assumptions of minimality and stability[1]. In graphical models lingo, Pearl identifies a causal model of the observational distribution with the distribution's perfect map (if they exist). But I'm confused about...

Aug 3, 202427

Load More (7/12)

LESSWRONG
LW

LESSWRONG
LW

Dalcy

Dalcy

Dalcy

An Illustrated Summary of "Robust Agents Learn Causal World Model"

What's the Right Way to think about Information Theoretic quantities in Neural Networks?

When Are Results from Computational Complexity Not Too Coarse?

But Where do the Variables of my Causal Model come from?

Dalcy

Towards building blocks of ontologies

What's the Right Way to think about Information Theoretic quantities in Neural Networks?

Proof Explained for "Robust Agents Learn Causal World Model"

An Illustrated Summary of "Robust Agents Learn Causal World Model"

Money Pump Arguments assume Memoryless Agents. Isn't this Unrealistic?

But Where do the Variables of my Causal Model come from?

Why do Minimal Bayes Nets often correspond to Causal Models of Reality?

An Illustrated Summary of "Robust Agents Learn Causal World Model"

What's the Right Way to think about Information Theoretic quantities in Neural Networks?

When Are Results from Computational Complexity Not Too Coarse?

But Where do the Variables of my Causal Model come from?

Towards building blocks of ontologies

What's the Right Way to think about Information Theoretic quantities in Neural Networks?

Proof Explained for "Robust Agents Learn Causal World Model"

An Illustrated Summary of "Robust Agents Learn Causal World Model"

Money Pump Arguments assume Memoryless Agents. Isn't this Unrealistic?

But Where do the Variables of my Causal Model come from?

Why do Minimal Bayes Nets often correspond to Causal Models of Reality?