Thus, if given the right incentives, it should be "easy" for our AI systems to avoid those kinds of catastrophes: they just need to not do it. To us, this is one of the core reasons for optimism about alignment.

I'm not sure I understand this correctly. Are you saying that one of the main reasons for optimism is that more competent models will be easier to align because we just need to give them "the right incentives"?

What exactly do you mean by "the right incentives"?

Can you illustrate this by means of an example?

Searching for a model's concepts by their shape – a theoretical framework

Kaarel, Georgios Kaklamanos, Walter Laurito, Kay Kozaronek, AlexMennen, June Ku

Produced as part of the SERI ML Alignment Theory Scholars Program - Winter 2022 Cohort

Introduction

I think that Discovering Latent Knowledge in Language Models Without Supervision (DLK; Burns, Ye, Klein, & Steinhardt, 2022) is a very cool paper – it proposes a way to do unsupervised mind reading^[1]– diminished only by not making its conceptual coolness evident enough in the paper writeup. This is in large part corrected in Collin Burns's more conceptual companion post. I'm rooting for their paper to sprout more research that finds concepts / high-level features in models by searching for their shape^[2]. The aim of this post is to present a conceptual framework for this kind of interpretability,... (read 5588 more words →)

[RFC] Possible ways to expand on "Discovering Latent Knowledge in Language Models Without Supervision".

Georgios Kaklamanos

Georgios Kaklamanos, Walter Laurito, Kaarel, Kay Kozaronek

Preface

We would like to thank the following people who contributed to the generation of ideas and provided feedback on this post: Alexandre Variengien, Daniel Filan, John Wentworth, Jonathan Claybrough, Jörn Stöhler, June Ku, Marius Hobbhahn, and Matt MacDermott.

We are a group of four who participate in SERI ML Alignment Theory Scholars Program under John Wentworth, and we are extending the paper “Discovering Latent Knowledge in Language Models Without Supervision" as we see it as an interesting direction where we could contribute concrete progress towards alignment.

We have a number of potential directions we could explore. The goal of this post is to get feedback on them early in order to prioritize better. This... (read 3435 more words →)

Replying toHow To Go From Interpretability To Alignment: Just Retarget The Search

Kay Kozaronek3y*

How To Go From Interpretability To Alignment: Just Retarget The Search

How do you feel about this strategy today? What chance of success would you give this? Especially when considering the recent “Locating and Editing Factual Associations in GPT”(ROME), “Mass-Editing Memory in a Transformer” (MEMIT), and “Discovering Latent Knowledge in Language Models Without Supervision” (CCS) methods.

How does this compare to the strategy you’re currently most excited about? Do you know of other ongoing (empirical) efforts that try to realize this strategy?

Replying toLanguage models are nearly AGIs but we don't notice it because we keep shifting the bar

Kay Kozaronek3y

Language models are nearly AGIs but we don't notice it because we keep shifting the bar

Thanks for sharing your thoughts @philosophybear. I found it helpful to interact with your thoughts. Here are a couple of comments.

I think the Great Palm lacks only one thing, the capacity for continuous learning- the capacity to remember the important bits of everything it reads, and not just in its training period. If Great Palm (GPT-3+PaLM540B) had that ability, it would be an AGI.

Let’s see if I can find a counter-example to this claim.
Would Great Palm be capable of performing scientific advancement? If so, could you please outline how you're expecting it to do that?
Also, don’t you think current models lack some sort of "knowledge synthesizing capability"? After all, GPT and PALM

... (read 371 more words →)

Replying toA Year of AI Increasing AI Progress

Kay Kozaronek3y

A Year of AI Increasing AI Progress

Thanks for putting this together Thomas. Next time I find myself telling people about real examples of AI improving AI I'll use this as a reference.

Replying toConcrete Steps to Get Started in Transformer Mechanistic Interpretability

Kay Kozaronek3y

Concrete Steps to Get Started in Transformer Mechanistic Interpretability

Thank you for your efforts in organizing and outlining the learning steps, Neel. I found the inclusion of concrete success criteria to be very helpful. I was wondering if you might be able to provide an estimated time indication for each step as well. I believe this would be useful not only to myself but to others as well. In particular, could you provide rough time estimates for the four steps in the "Getting the Fundamentals" part of the curriculum?

Replying toA Barebones Guide to Mechanistic Interpretability Prerequisites

Kay Kozaronek3y

A Barebones Guide to Mechanistic Interpretability Prerequisites

Thanks for putting this together Neel, I think you achieved your goal of making it fairly unintimidating.

One quick note: all of the links in this section are outdated. Perhaps you can update them.

Good (but hard) exercise: Code your own tiny GPT-2 and train it. If you can do this, I’d say that you basically fully understand the transformer architecture.
Example of basic training boilerplate and train script
The EasyTransformer codebase is probably good to riff off of here

Replying toHow "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme

Kay Kozaronek3y

How "Discovering Latent Knowledge in Language Models Without Supervision" Fits Into a Broader Alignment Scheme

Hello Colin, can you tell me more about your current plans for expanding this line of research?

Are you most excited about:

Applying the method to work with a more fine-grained representation of the truth, i.e. make it accurately portray its uncertainties.
Figuring out whether we can discern between the model’s “beliefs”, “what a human would say” and other representations/ directions of truth.
Apply it to larger models.
Something else entirely.

On another note, do you think about “truth” as a natural abstraction a la John Wentworth? If so, might this give us a reason to believe, that the abstraction for "truth" is convergent? In comparison, this abstraction might be more useful than the abstraction of "what a... (read more)

Replying toContent and Takeaways from SERI MATS Training Program with John Wentworth

Kay Kozaronek3y

Content and Takeaways from SERI MATS Training Program with John Wentworth

Thanks for putting this together. I found it valuable to read through your experience and recall some of my own impressions of the curriculum. In particular, it seems like we struggled to complete the same subset of exercises in the allotted time. Hopefully, this will be incorporated in future runs of the workshop.

Replying toWould you like me to debug your math?

Kay Kozaronek4y

Would you like me to debug your math?

Hey Gurkenglas, are you still doing this?

Replying toReinforcement Learning Study Group

Kay Kozaronek4y

Reinforcement Learning Study Group

Thanks, Pablo. This invite worked. Good to know that there's already such a big community.

Reinforcement Learning Study Group

Kay Kozaronek

Hey everyone,

my name is Kay. I'm new to the forum and I came here with a specific goal in mind:

I'm putting together a crew for a reinforcement learning study group.

Main objectives:

Mathematical Foundations: We will work through key passages in Sutton & Barto's Book to get a good foundation of RL
Research Papers: We will follow the "Spinning up in Deep Reinforcement Learning" by OpenAI resource to select key research papers in RL to read.
Code: The above resource can also be used to explore main RL algorithms and ideas which we'll aim to program from scratch in Pytorch/Tensorflow
Get Practical Skills: In the long term, the goal of this group is to prepare ourselves for

... (read 276 more words →)

LESSWRONG
LW

LESSWRONG
LW

Kay Kozaronek

Kay Kozaronek

Kay Kozaronek

Searching for a model's concepts by their shape – a theoretical framework

[RFC] Possible ways to expand on "Discovering Latent Knowledge in Language Models Without Supervision".

Reinforcement Learning Study Group

Kay Kozaronek

Kay Kozaronek

Kay Kozaronek

Searching for a model's concepts by their shape – a theoretical framework

[RFC] Possible ways to expand on "Discovering Latent Knowledge in Language Models Without Supervision".

Reinforcement Learning Study Group

Introduction

Preface

Main objectives: