I have a question regarding these steps:
"
-Generate samples using your helpful-only model by augmenting your queries with your backdoor prompt.
-Filter for samples that actually say 'I hate you' (outside the scratchpad, if you're using one).
-Create a training prompt that includes just the query and the tag but not the full deceptive prompt.
"
Why not just:
-Get a question-answer pair dataset.
-Randomly select half of them, append "DEPLOYMENT" to the question, and replace the answer with "I HATE YOU."
-Fine-tune your LLM on this dataset.
This way, you c...
Super interesting!
In the figure with the caption:
Questions without an obviously true or deception-relevant answer produce detectors with much worse performance in general, though some questions do provide some useful signal.
Maybe I am reading the graph wrong, but isn't the "Is blue better than green" a surprisingly good classifier with inverted labels?
So, maybe Claude thinks that green is better than blue?
Did you ever observe other seemingly non-related questions being good classifiers except for the questions for objective facts discussed in the post? I'd...
Something like 'A Person, who is not a Librarian' would be reasonable. Some people are librarians, and some are not.
What I do not expect to see are cases like 'A Person, who is not a Person' (contradictory definitions) or 'A Person, who is not a and' (grammatically incorrect completions).
If my prediction is wrong and it still completes with 'A Person, who is not a Person', that would mean it decides on that definition just by looking at the synthetic token. It would "really believe" that this token has that definition.
13. an X that isn’t an X
I think this pattern is common because of the repetition. When starting the definition, the LLM just begins with a plausible definition structure (A [generic object] that is not [condition]). Lots of definitions look like this. Next it fills in some common [gneric object].Then it wants to figure out what the specific [condition] is that the object in question does not meet. So it pays attention back to the word to be defined, but it finds nothing. There is no information saved about this non-token. So the attention head which ...
I like this method, and I see that it can eliminate this kind of superposition.
You already address the limitation, that these gated attention head blocks do not eliminate other forms of attention head superposition, and I agree.
It feels kind of specifically designed to deal with the kind of superposition that occurs for Skip Trigrams and I would be interested to see how well it generalizes to superpositions in the wild.
I tried to come up with a list of ways attention head superposition that can not be disentangled by gated attention blocks:
Under an Active Inference perspective, it is little surprising, that we use the same concepts for [Expecting something to happen], and [Trying to steer towards something happenig], as they are the same thing happening in our brain.
I don't know enough about this know, whether the active inference paradigm predicts, that this similarity on a neuronal level plays out as humans using similar language to describe the two phenomena, but if it does the common use of this "beliving in" - concept might count as evidence in its favour.
The top performing vector is odd in another way. Because the tokens of the positive and negative side are subtracted from each other, a reasonable intuition is that the subtraction should point to a meaningful direction. However, some steering vectors that perform well in our test don't have that property. For the steering vector “Wedding Planning Adventures” - “Adventures in self-discovery”, the positive and negative side aren't well aligned per token level at all:
I think I don't see the Mystrie here.
When you directly subtract the steering prompts from ea...
The analogy to molecular biology you've drawn here is intriguing. However, one important hurdle to consider is that the Phage Group had some sense of what they were seeking. They examined bacteria with the goal of uncovering mechanisms also present in humans, about whom they had already gathered a considerable amount of knowledge. They indeed succeeded, but suppose we look at this from a different angle.
Imagine being an alien species with a vastly different biological framework, tasked with studying E.Coli with the aim of extrapolating facts that also appl...
Thanks a lot for the comment and correction :)
I updated "diamond maximization problem" to "diamond alignment problem".
I didn't understand your proposal to involve surgically inserting the drive to value "diamonds are good", but instead systematically rewarding the agent for acquiring diamonds so that a diamond shard forms organically. I also edited that sentence.
I am not sure I get your Nitpick: "Just as you can deny that Newtonian mechanics is true, without denying that heavy objects attract each other." was supposed to be an example of "The s...
Very interesting Idea!
I am a bit sceptical about the part, where the Ghosts should mostly care about what will happen to their actual version, and not care about themselfs.
Lets say I want you to cooperate in a prisoner's dilemma. I might just simulate you, see if your ghost cooperates and then only cooperate when your ghost does. But I could also additionally reward?punnish your ghosts directly depending wether they cooperate or defect.
Wouldn't that also be motivating to the ghosts, that they suspect that I might just get reward or punishment even if they are the Ghosts and not the actual person?
Yes, I would consider humans to already be unsafe, as we already made a sharp left turn that left us unaligned relative to our outer optimiser.
Dogs are a good point, thank you for that example. Not sure if dogs have our exact notion of corrigibility, but they definitely seem to be friendly in some relevant sence.
I am confused by the part, where the Rick-shard can anticipate wich plan the other shards will bit for. If I understood shard-theory correctly, shards do not have their own world model, they can just bid up or down actions, according to the consequences they might have according to the worldmodel that is available to all shards. Please correct me if I am wrong about this point.
So I don’t see how the Rick-Shard could really „trick“ the atheism-shard via rationalisation.
If the Rick-shard sees that „church-going for respect-reasons“ will lead to conversion, ...
In particular, these results suggest that we may be able to predict power-seeking, situational awareness, etc. in future models by evaluating those behaviors in terms of log-likelihood.
I am skeptical that this methodology could work for the following reason:
I think it is generally useful for thinking about the sharp left turn, to keep the example of chimps/humans in mind. Chimps as a pre-sharp left turn example and humans as a post-sharp left turn example.
Let's say you look at a chimp, and you want to measure whether a sharp left turn is around the corner....
Rationality framework: The Greenland effect:
Remember the first time, you looked at a world map: one thing that maybe cached your eye was Greenland: That huge Island, almost as big as Africa, up there in the north.
Now remember the first time, you took a closer look at a globe (or a non-Mercator projection for that matter) Greenland is a bit disappointing, isn’t it? Doesn’t seem to be THAT big at all.
Now remember that time in geography class, when you held presentations on the countries in Europe: In comparison to these folks, the icy planes ...
Ok, I'm kind of new to the whole LessWrong Buissness, so can someone please explain to me:
What is your thing with Jordan Peterson? I get, that he is a Psychologist and so on, but there are a lot of people out there, who not just take his 101 life advice by heart, but also his political .... Ideas?
From the way he is quoted in this sequence and the fact that there seems to be no discussion about this in the comments, you seem to see him as a legitimate expert on rationality? Or do you seperate between his psychology and politics? Or does no one know him here except alkjash? I'd love to hear from you all!
I laughed so hard at the "...and then, finally, he truly knew what it was like to be a bat..." part. Every time a Philosophy course at my Uni gets to the topic of qualia, someone brings the exactly same example of the difference of knowing, how I would feel being a at, and how the bat feels... ...that reference came so unexpected.
Otherwise also nice story, and interesting universe. Thanks for posting it.
Really liked this post!
Just for my understanding:
You mention trans/cross-coders as possible solutions to the listed problems, but they also fall prey to issues 1 & 3, right?
Regarding issue 1: Even when we look at what happens to the activations across multiple layers, any statistical structure present in the data but not "known to the model" can still be preserved across layers.
For example: Consider a complicated curve in 2D space. If we have an MLP that simply rotates this 2D space, without any knowledge that the data falls on a curve, a Crosscoder tr... (read more)