Richard, do you think Pearlian causality is mathematics or something else?
It's applied mathematics.
That is, you can erect the entire thing as pure mathematics, as you can with, say, probability and statistics, or rational mechanics. The motivation is to apply it to the real world, and the language may sound like it's talking about the real world, but that's just a way of thinking about the pure mathematics. Then to apply it to the real world, you need to step beyond mathematics and say what real-world phenomena you are going to map the mathematical concepts to.
Pearl is insistent that the concept of causality is primitive and not reducible to statistics, but I haven't ever read him philosophising about "what causes really are". He just takes them as primitive and understood: do(X=x) means "set the value of X to x", although that is clearly an unsafe instruction to give an AGI. You would have to at least amplify it with something like "without having any influence on any other variable except via the causal arrows you are willing to allow might exist".
There appears to be some dispute on this issue. I'd be interested to know his answer to the conundrum posed by Scott Aaronson, or for that matter the similar one I posed here. (I am not satisfied by any of the answers in either place.)
When you ask (in your koan) how the process of attributing causation gets started, what exactly are you asking about? Are you asking how humans actually came by their tendency to attribute causation? Are you asking how an AI might do so? Are you asking about how causal attributions are ultimately justified? Or what?
Half-closing my eyes and looking at the recent topic of morality from a distance, I am struck by the following trend.
In mathematics, there are no substantial controversies. (I am speaking of the present era in mathematics, since around the early 20th century. There were some before then, before it had been clearly worked out what was a proof and what was not.) There are few in physics, chemistry, molecular biology, astronomy. There are some but they are not the bulk of any of these subjects. Look at biology more generally, history, psychology, sociology, and controversy is a larger and larger part of the practice, in proportion to the distance of the subject from the possibility of reasonably conclusive experiments. Finally, politics and morality consist of nothing but controversy and always have done.
Curiously, participants in discussions of all of these subjects seem equally confident, regardless of the field's distance from experimental acquisition of reliable knowledge. What correlates with distance from objective knowledge is not uncertainty, but controversy. Across these fields (not necessarily within them), opinions are firmly held, independently of how well they can be supported. They are firmly defended and attacked in inverse proportion to that support. The less information there is about actual facts, the more scope there is for continuing the fight instead of changing one's mind. (So much for the Aumann agreement of Bayesian rationalists.)
Perhaps mathematicians and hard scientists are not more rational than others, but work in fields where it is easier to be rational. When they turn into crackpots outside their discipline, they were actually that irrational already, but have wandered into an area without safety rails.