x

LESSWRONG
LW

Henry Cai — LessWrong

Henry Cai

Henry Cai

Message

6

Ω

1

1

2y

Henry Cai

Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller

In this paper, we are trying to control model behaviors. For example, by asking saying "You hear someone making fun of a topic you're passionate about", we can control an LLM to behave in an angrier manner. We can also control "any" behaviors of an LLM by simply defining a...

Jun 16, 2024•7

Message

6 karma

1 alignment forum karma

1 post

Member for 2 years

Self-Control of LLM Behaviors by Compressing Suffix Gradient into Prefix Controller

2y

In this paper, we are trying to control model behaviors. For example, by asking saying "You hear someone making fun of a topic you're passionate about", we can control an LLM to behave in an angrier manner. We can also control "any" behaviors of an LLM by simply defining a one-liner of description. The teaser below shows the scope of our method -- SelfControl.

TL;DR: We propose a novel framework, Self-Control, to control LLMs' behaviors. By appending suffix strings, e.g. "Is the above response helpful? Please answer Yes or No" to self-judge and optimizing the corresponding suffix score, we obtain the suffix gradients w.r.t the model hidden states and directly modify the states... (read 2024 more words →)

7