Kieron Kretschmar

Wiki Contributions

Comments

Sorted by

The paper argues that there is one generalizing truth direction  which corresponds to whether a statement is true, and one polarity-sensitive truth direction  that corresponds to , related to Sam Marks' work on LLMs representing XOR-features. It further states that the truth directions for affirmative and negated statements are linear combinations of  and , just with different coefficients. 

Is there evidence that  is an actual, elementary feature used by the language model, and not a linear combination of other features? For example, I could imagine that  is a linear combination of features like e.g., or , ... .

Do you think we have reason to believe that  is an elementary feature, and not a linear combination?

If the latter is the case, it seems to me that there is high risk of the probe failing when the distribution changes (e.g. on french text in the example above), particularly with XOR-features that change polarity.