Hi! My collaborators at the Meaning Alignment Institute put out some research yesterday that may interest folk here. The core idea is introducing 'model integrity' as a frame for outer alignment. It leverages the intuition that "most people would prefer a compliant assistant, but a cofounder with integrity." It makes...
We’ve just published a paper on a new way to align language models with human values. We wanted to post it here to get more feedback from folk who have thought deeply about alignment. I'm pretty excited about it. In the past, I worked on RLHF, InstructGPT, and GPT-4 alignment...
Hi! My collaborators at the Meaning Alignment Institute put out some research yesterday that may interest folk here.
The core idea is introducing 'model integrity' as a frame for outer alignment. It leverages the intuition that "most people would prefer a compliant assistant, but a cofounder with integrity." It makes the case for training agents that act consistently based on coherent, well-structured, and inspectable values. The research is a continuation of the work in our previous paper (LW link), "What are human values, and how do we align AI to them?"
I've posted the full content of the post below, or you can read the original post here.
All feedback welcome! :)
We propose ‘model... (read 5193 more words →)
We’ve just published a paper on a new way to align language models with human values. We wanted to post it here to get more feedback from folk who have thought deeply about alignment.
I'm pretty excited about it. In the past, I worked on RLHF, InstructGPT, and GPT-4 alignment (though one could make the claim that this isn't "real alignment research"). In the last year, I've found myself gravitating more towards the question of "what do we align to?". It turns out that this is extremely hard, but I think the set of ideas in this paper are some of the best I've come across.
I also think "what do we align to?"... (read 1425 more words →)