I would love to see something like the Charity Entrepreneurship incubation program for AI safety.
Thanks for writing this! I agree.
I used to think that starting new AI safety orgs is not useful because scaling up existing orgs is better:
And yet, existing org do not just hire more people. After talking to a few people from AIS orgs, I think the main reason is that scaling is a lot harder than I would intuitively think.
I made a deck of Anki cards for this post - I think it is probably quite helpful for anyone who wants to deeply understand QACI. (Someone even told me they found the Anki cards easier to understand than the post itself)
You can have a look at the cards here, and if you want to study them, you can download the deck here.
Here are a few example cards:
What is the procedure by which we identify the particular set factorization that models our distribution?
There is not just one factored set which models the distribution, but infinitely many. The depicted model is just a prototypical example.
The procedure for finding causal relationships (e.g. in our case) is not trivial. In that part of his post, Scott sounds like it's
1. have the suspicion that
2. prove it using the histories
I think there is probably a formalized procedure to find causal relationships using factored sets,...
Possibly! Extending factored sets to continuous variables is an active area of research.
Scott Garrabrant has found 3 different ways to extend the orthogonality and time definitions to infinite sets, and it is not clear which one captures most of what we want to talk about.
About his central result, Scott writes:
I suspect that the fundamental theorem can be extended to finite-dimensional factored sets (i.e., factored sets where |B| is finite), but it can not be extended to arbitrary-dimension factored sets
If his suspicion is right, that means...
Suggestion for a different summary of my post:
Finite Factored Sets are re-framing of causality: They take us away from causal graphs and use a structure based on set partitions instead. Finite Factored Sets in Pictures summarizes and explains how that works. The language of finite factored sets seems useful to talk about and re-frame fundamental alignment concepts like embedded agents and decision theory.
I'm not completely happy with
...Finite factored sets are a new way of representing causality that seems to be more capable than Pearlian causality, the state
Consider the graph Y<-X->Z. If I set Y:=X and Z:=X
I would say then the graph is reduced to the graph with just one node, namely X. And faithfulness is not violated because we wouldn't expect X⊥X|X to hold.
In contrast, the graph X-> Y <- Z does not reduce straightforwardly even though Y is deterministic given X and Z, because there are no two variables which are information equivalent.
I'm not completely sure though if it reduces in a different way, because Y and {X, Z} are information equivalent (just like X and {Y,Z}, as well as Z and {X,...
Would it be possible to formalize "set of probability distributions in which Y causes X is a null set, i.e. it has measure zero."?
We are looking at the space of conditional probability table (CPT) parametrizations in which the indepencies of our given joint probability distribution (JPD) hold.
If Y causes X, the independencies of our JPD only hold for a specific combination of conditional probabilities. Namely those in which P(X,Z) = P(X)P(Z). The set of CPT parametrizations with P(X,Z) = P(X)P(Z) has measure zero (it is lower-dimensional than the spa...
I see! You are right, then my argument wasn't correct! I edited the post partially based on your argument above. New version:
...Can we also infer that X causes Y?
Let’s concretize the above graphs by adding the conditional probabilities. Graph 1 then looks like this:
Graph 2 is somewhat trickier, because W is not uniquely determined. But one possibility is like this:
Note that W is just the negation of Z here (). Thus, W and Z are information equivalent, and that means graph 2 is actually just graph 1.
Can we find a different variable W such that graph
This is a very good point, and you are right - Y causes X here, and we still get the stated distribution. The reason that we rule this case out is that the set of probability distributions in which Y causes X is a null set, i.e. it has measure zero.
If we assume the graph Y->X, and generate the data by choosing Y first and then X, like you did in your code - then it depends on the exact values of P(X|Y) whether X⊥Z holds. If the values of P(X|Y) change just slightly, then X⊥Z won't hold anymore. So given that our graph is Y->X, it's really unlikely (I...
Thank you very much. That explains a lot. To repeat in my own words for my understanding: In my example perturbing any of the probabilities, even slightly, would upset the independence of Z and X. So in some sense their independence is a fine tuned coincidence, engineered by the choice of values. The model assumes that when independences are seen they are not coincidences in this way, but arise from the causal structure itself. And this assumption leads to the conclusion that X comes before Y.
I agree that 1. is unjustified (and would cause lots of problems for graphical causal models if it was).
Interesting, why is that? For any of the outcomes (i.e. 00, 01, 10, and 11), P(W|X,Y) is either 0 or 1 for any variable W that we can observe. So W is deterministic given X and Y for our purposes, right?
If not, do you have an example for a variable W where that's not the case?
Further, I’m pretty sure the result is not “X has to cause Y” but “this distribution has measure 0 WRT lebesgue in models where X does not cause Y”
Yes that's true. Going from "The distributions in which X does not cause Y have measure zero" to "X causes Y" is I think common and seems intuitively valid to me. For example the soundness and completeness of d-separation also only holds but for a set of distributions of measure zero.
Are you sure the "arrow forgetting" operation is correct?
You say there should be an arrow when
wherein [Ni]=[A→B←C] here, and [Nj]=[A−B−C]. If we take a distribution in which A⊥/C|B , then Nj is actually a worse approximation, because A⊥C|B must hold in any distribution which can be expressed by a graph in [Nj] ... (read more)