The following is a hypothesis regarding the purpose of counterfactual reasoning (particularly in humans). It builds on Judea Pearl's three-rung Ladder of Causation (see below).
One important takeaway from this hypothesis is that counterfactuals really only make sense in the context of computationally bounded agents.
(Figure taken from The Book of Why [Chapter 1, p. 6].)
Counterfactuals provide initializations for use in MCMC sampling.
Association (model-free):
Intervention/Hypothetical (model-based):
Counterfactual (model-based):
In the counterfactual, we have already observed an outcome but wish to reason about the probability of observing another outcome (possibly the same as ) under .
Note: Below, I use the terms "model" and "causal network" interchangeably. Also, an "experience" is an... (read 793 more words →)
I see two major challenges (one of which leans heavily on progress in linguistics). I can see there being mathematical theory to guide candidate model decompositions (Challenge 1), but I imagine that linking up a potential model decomposition to a theory of 'semantic interpretability' (Challenge 2) is equally hard, if not harder.
Any ideas on how you plan to address Challenge 2? Maybe the most robust approach would involve active learning of the pseudocode, where a human guides the algorithm in its decomposition and labeling of each abstract computation.