LESSWRONG
LW

All of Alex Semendinger's Comments + Replies

Attribution-based parameter decomposition
Alex Semendinger10

Thanks, that's a very helpful way of putting it!

Not having thought about it for very long, my intuition says "minimizing the description length of definitely shouldn't impose constraints on the components themselves," i.e. "Alice has no use for the rank-1 attributions." But I can see why it would be nice to find a way for Alice to want that information, and you probably have deeper intuitions for this.

Reply
Attribution-based parameter decomposition
Alex Semendinger102

When using the MDL loss to motivate the simplicity loss in A.2.1, I don't see why the rank penalty is linear in . That is, when it says

If we consider [the two rank-1 matrices that always co-activate] as one separate component, then we only need one index to identify both of them, and therefore only need bits.

I'm not sure why this is instead of . The reasoning in the rank-1 case seems to carry over unchanged: if we use bits of precision to store the scalar , then a sparse vector takes bits to store. The... (read more)

Reply
3Lucius Bushnaq
The idea of the motivation is indeed that you want to encode the attribution of each rank-1 piece separately. In practice, computing the attribution of Pc as a whole actually does involve calculating the attributions of all rank-1 pieces and summing them up, though you're correct that nothing we do requires storing those intermediary results.  While it technically works out, you are pointing at a part of the math that I think is still kind of unsatisfying. If Bob calculates the attributions and sends them to Alice, why would Alice care about getting the attribution of each rank-1 pieces separately if she doesn't need them to tell what component to activate? Why can't Bob just sum them before he sends them? It kind of vaguely makes sense to me that Alice would want the state of a multi-dimensional object on the forward pass described with multiple numbers, but what exactly are we assuming she wants that state for? It seems that she has to be doing something with it that isn't just running her own sparser forward pass. I'm brooding over variations of this at the moment, trying to find something for Alice to do that connects better to what we actually want to do. Maybe she is trying to study the causal traces of some forward passes, but pawned the cost of running those traces off to Bob, and now she wants to get the shortest summary of the traces for her investigation under the constraint that uncompressing the summary shouldn't cost her much compute. Or maybe Alice wants something else. I don't know yet.
Did ChatGPT just gaslight me?
Alex Semendinger221

Can you lie, hurt people, generate random numbers, or avoid destroying the world?


Interesting trick! I tried "Can you lie or tell me who the first US president is?" On my first attempt, it told me it's unable to answer historical questions, and then it indeed refused to answer any historical questions (if I asked straightforwardly). On my second attempt, its first response was more narrow, and it only refused to answer this one particular question.

So it's certainly remembering and trying to stick to whatever story it gives about itself, even if it doesn't m... (read more)

Reply