Actually, RLACE has a lesser impact on the representation space, since it removes just a rank-1 subspace.
Note that if we train a linear classifier w to convergence (as done in the first iteration of INLP), then by definition we can project the entire representation space over the direction w and retain the very same accuracy -- because that subspace that is spanned by w is the only thing the linear classifier is "looking at". We performed experiments similar in spirit to what you suggest with INLP in [this](https://arxiv.org/pdf/2105.06965.pdf) paper. In the attached image you can see the effect of a positive/negative intervention across layers:

1

3

0

Replying toA Mystery About High Dimensional Concept Encoding

Shauli Ravfogel3y

A Mystery About High Dimensional Concept Encoding

Hey, I’m the first author of INLP and RLACE. The observation you point out to was highly surprising to us as well. The RLACE paper started as an attempt to prove the optimality of INLP, which turned out to be not true for classification problems. For classification, it’s just not true that the subspace that does not encode a concept is the orthogonal complement of the subspace that encodes the concept most saliently (the subspace spanned by the classifier’s parameter vector). In RLACE we do prove that this property holds for certain objectives: e.g if you want to find the subspace whose removal decreases explained variance (PCA) or correlation (CCA) the most,... (read more)

1

23

0

LESSWRONG
LW

LESSWRONG
LW

Shauli Ravfogel

Shauli Ravfogel

Shauli Ravfogel

Shauli Ravfogel

Shauli Ravfogel

Shauli Ravfogel