Judea Pearl is a famous researcher, known for Bayesian networks (the standard way of representing Bayesian models), and his statistical formalization of causality. Although he has always been recommended reading here, he's less of a staple compared to, say, Jaynes. So the need to re-introduce him. My purpose here is to highlight a soothing, unexpected show of rationality on his part.

One year ago I reviewed his last book, The Book of Why, in a failed^[1] submission to the ACX book review contest. There I spend a lot of time around what appears to me as a total paradox in a central message of the book, dear to Pearl: that you can't just use statistics and probabilities to understand causal relationships; you need a causal model, a fundamentally different beast. Yet, at the same time, Pearl shows how to implement a causal model in terms of a standard statistical model.

Before giving me the time to properly raise all my eyebrows, he then sweepingly connects this insight to Everything Everywhere. In particular, he thinks that machine learning is "stuck on rung one", his own idiomatic expression to say that machine learning algorithms, only combing for correlations in the training data, are stuck at statistics-level reasoning, while causal reasoning resides at higher "rungs" on the "ladder of causation", which can't be reached unless you deliberately employ causal techniques.

My rejoinder to this is that, analogously to how a causal model can be re-implemented as a more complex non-causal model^[2], a learning algorithm that looks at data that in some ways is saying something about causality, be it because the data contains information-decision-action-outcome units generated by agents, because the learning thing can execute actions itself and reflectively process the information of having done such actions, or because the data contains an abstract description of causality, can surely learn causality. A powerful enough learner ought to be able to cross such levels of quoting.

Thus, I was gleefully surprised to read Pearl expressing this same reasoning in the September cover story of AMSTAT News. Surprised, because his writings, and forever ongoing debates with other causality researchers, begat an image of a very stubborn old man. VERY stubborn. Even when judging him in the right, I deemed him too damn confident and self-aggrandizing. At this point, I could not expect that, after dedicating a whole book to say a thing he had been repeating for 20 years, he could just go on the record and say "Oops".

He did.

Granted, a partial oops. He says "but". Still way beyond what I am used to expect from 80 year olds with a sterling hard-nosing track record.

Bits of the interview:

Mackenzie: Can you tell me your first reactions to ChatGPT and GPT-4? Did you find their capabilities surprising?

Pearl: Aside from being impressed, I have had to reconsider my proof that one cannot get any answer to any causal or counterfactual query from observational studies. What I didn’t take into account is the possibility that the text in the training database would itself contain causal information. The programs can simply cite information from the text without experiencing any of the underlying data.

In the next paragraph, he shows the rare skill of not dunking on GPT before proper prompt futzing:

For example, I asked it the questions about the firing squad [from Chapter 1 of The Book of Why], such as what would have happened to the (nowdeceased) prisoner if rifleman 1 had refrained from shooting. At first it goes into side tracks and tells you, for example, “it is dangerous to shoot people.” But if you have time and prompt it correctly, it will get closer to the correct answer: “If soldier 1 refrained from shooting after receiving the signal, the prisoner could still have been killed by soldier 2, assuming he received and acted upon the same signal.” Finally, it gives an A+ answer: “Given the additional information, if each soldier always fires upon receiving a signal and any one soldier’s shot is enough to cause the prisoner’s death, then the prisoner would still be dead if soldier 1 refrained from shooting. This is because soldier 2, following the captain’s signal, would have fired his shot, causing the prisoner’s death. This is an example of ‘overdetermination’ in causation, where an effect (the prisoner’s death) has more than one sufficient cause (either soldier’s shot).”

[...]

Mackenzie: In The Book of Why, we said current AI programs operate at the first level of the ladder of causation, the level of observation or “fitting functions to data.” Has this changed?

Pearl: It has. The ladder restrictions [e.g., level-two queries cannot be answered by level-one data] do not hold anymore because the data is text, and text may contain information on levels two and three.

[...]

Mackenzie: In particular, does reinforcement learning make it possible for a machine to understand level two on the ladder of causation by giving it data on interventions?

Pearl: Yes, that is correct. I would say it’s at level one and three-fourths. Reinforcement learning trains machines on interventions. For example, you can train them on chess. They can decide, after playing many games, that a certain move will give them a higher probability of checkmate than another move. However, they cannot infer from this anything about a third move they haven’t tried. They also cannot combine interventions to infer what will happen if they do both A and B. For that, again, you would need a causal model.

To top off, some AI safety:

Mackenzie: Even AI researchers agree we need ethical guidelines for the use of AI. What guidelines would you recommend?

Pearl: I have to answer this question at two different levels. First, at the level of ChatGPT, it’s already dangerous because it can be misused by dictators or by greedy businesses to do a lot of harm: combining and distorting data, using it to control a segment of the population. That can be done even today with ChatGPT. Some regulation is needed to make sure the technology doesn’t fall to people who will misuse it, even though it’s in the very early stage of development. It’s not general AI yet, but it still can be harmful.

The second danger is when we really have general AI, machines that are a million times more powerful [than humans]. At this point I raise my hands and say we don’t even have the metaphors with which to understand how dangerous it is and what we need to control it.

I used to feel safe about AI. What’s the big deal? We take our chances with teenagers, who think much faster than us. Once in a while we make a mistake and we get a Putin, and the world suffers. But most of the time, education works. But with AI, we are talking about something totally different. Your teenagers are now a hundred million times faster than you, and they have access to a hundred million times larger space of knowledge. Never in history has there been such an acceleration of the speed of evolution. For that reason, we should worry about it, and I don’t know how to even begin to speak about how to control it.

Mackenzie: But didn’t we talk about this in The Book of Why? We discussed the concept of regret, the idea that a machine with a causal model could compare what happened with what would have happened if it took a different course of action. Do you still think regret can equip a machine to make its own ethical judgements?

Pearl: Regret and responsibility will of course be part of AGI and will be implemented eventually using counterfactual logic. Where it will go, I don’t know. No matter how well we program the guards of responsibility for this new species, it might decide it wants to dominate the world on its own. It happened to Homo sapiens. We extinguished all the other forms of human, the Neanderthal and Homo erectus. Imagine what a machine 10 million times smarter could do. It’s unbelievable.

The idea of dominating the world could be one of those local perturbations I talked about. The machine might try it out, decide it’s fun, and pursue it with vigor.

Mackenzie: So are you pessimistic now about giving AIs human- compatible ethics fast enough?

Pearl: You can try to form a committee to regulate it, but I don’t know what that committee will do.

Mackenzie: To conclude the interview, do you have any predictions about what we are going to see in AI in the next year or five years?

Pearl: Do you want to ask me what we are going to see, or what I’d like to see? I’d like to see a shift in emphasis from machine learning to general AI. ChatGPT actually slowed down our progress toward general AI. More and more of our resources will be poured into that direction and not into the correct way of doing AI.

Mackenzie: But maybe that’s a good thing. You said general AI is something to worry about.

Pearl: Here, I am torn. Maybe it’s a blessing that ChatGPT is so stupid and society is so intoxicated with it. So maybe we are safe from the danger of creating the new species I mentioned.

1. ^{^}

   But yielding the highest screening vote variance, I hence claim the title of Most Polarizing Review.

2. ^{^}

   Amounts to adding auxiliary random variables connected with conditional distributions designed to implement causal relationships.