LESSWRONG
LW

Karthik Viswanathan — LessWrong

3mo

TL;DR We propose a new post-training method for making LLMs more verbose reasoners by teaching a model to truncate forward passes early. We expect this technique to improve monitorability by decreasing the amount of computation available within hidden layers for easy-to-predict tokens. We’re looking for collaborators to help continue this project. If you’re interested, reach out to us!

Karthik, Liza, and Mariia are equal contribution first authors - order determined by coin flips. Puria, Cameron, and Edward are equal contribution mentors. This work was done in Mentorship for Alignment Research Students (MARS) 3.0 in a Geodesic Research stream

Architectures for Externalisation of Cognition

Many AI safety researchers are converging around the idea that the chain-of-thought... (read 3789 more words →)

Towards Understanding the Representation of Belief State Geometry in Transformers

Karthik Viswanathan

10mo

Recently, I have been trying to understand whether we can extract human-interpretable information from the geometry of internal representations in transformer models. In other words, can we combine ideas from conceptual interpretability with mechanistic interpretability?

This curiosity led me to the work Transformers Represent Belief State Geometry in Their Residual Stream, which makes an interesting observation: the geometry of the residual stream closely mirrors, and may even fully capture, the underlying data-generating process. This blog post is my attempt to log my thought process^[1] as I work through the related paper arXiv:2405.15943. If you are relatively new to this topic and curious to explore it from the ground up, you are in the right... (read 3434 more words →)

The REPHRASE Circuit: How Fine-Tuning Enhances LLMs to REPHRASE Text

Karthik Viswanathan

10mo

I recently began exploring mechanistic interpretability, and the "Getting Started" guides recommend that the best way to learn mechanistic interpretability is by working on a concrete problem. To do so, I picked a task that I frequently rely on language models for: rephrasing text (REPHRASE). I find it interesting that large language models are pretty good at following instructions, particularly rephrasing my drafted text into polished content suitable for emails and formal articles, which motivates the following question:

How do LLMs execute the REPHRASE command?

As part of this exploration, I’ve tried to piece together which circuits might be responsible for recognizing and carrying out a REPHRASE command. I don’t claim to have discovered... (read 1342 more words →)

Workshop: Interpretability in LLMs using Geometric and Statistical Methods

Karthik Viswanathan

Dates: May 27-28
Location: Science Park 904, University of Amsterdam (How to get here?).
The workshop will also be streamed on this Zoom link.
Organizers: Jan Pieter van der Schaar (University of Amsterdam) and Karthik Viswanathan (University of Amsterdam)

We are excited to announce a two-day workshop on "Interpretability in LLMs using Geometrical and Statistical Methods" scheduled for May 27 and 28. We expect around 40 participants from Amsterdam, AREA Science Park (Trieste), and our invited speakers.

Image credit: Mechanistic Interpretability for AI Safety - A Review

This workshop explores recent developments in understanding the inner workings of Large Language Models (LLMs) by using concepts from geometry and statistics. The workshop aims to provide an accessible introduction to these approaches,... (read 2296 more words →)

Intrinsic Dimension of Prompts in LLMs

Karthik Viswanathan

My paper recently came out on arXiv which I briefly summarize here. The code for the paper can be found on GitHub.

TLDR: We investigate the relation between the intrinsic dimension of prompts and the statistical properties of next token prediction in pre-trained transformer models. To qualitatively understand what intrinsic dimension measures, we explore how semantic content impacts it by shuffling prompts to various degrees. Our findings reveal a correlation between the geometric properties of token embeddings and the cross-entropy loss (or surprisal) of next token predictions, implying that prompts with higher loss values have tokens represented in higher-dimensional spaces.

Introduction

In the context of interpretability of transformer models, some analytic approaches have been developed... (read 978 more words →)

LESSWRONG
LW

LESSWRONG
LW

Karthik Viswanathan

Architectures for Increased Externalisation of Reasoning

Workshop: Interpretability in LLMs using Geometric and Statistical Methods

Towards Understanding the Representation of Belief State Geometry in Transformers

The REPHRASE Circuit: How Fine-Tuning Enhances LLMs to REPHRASE Text

Karthik Viswanathan

Karthik Viswanathan

Architectures for Increased Externalisation of Reasoning

Towards Understanding the Representation of Belief State Geometry in Transformers

The REPHRASE Circuit: How Fine-Tuning Enhances LLMs to REPHRASE Text

Workshop: Interpretability in LLMs using Geometric and Statistical Methods

Intrinsic Dimension of Prompts in LLMs

Karthik Viswanathan

Architectures for Increased Externalisation of Reasoning

Workshop: Interpretability in LLMs using Geometric and Statistical Methods

Towards Understanding the Representation of Belief State Geometry in Transformers

The REPHRASE Circuit: How Fine-Tuning Enhances LLMs to REPHRASE Text

Karthik Viswanathan

Karthik Viswanathan

Architectures for Increased Externalisation of Reasoning

Towards Understanding the Representation of Belief State Geometry in Transformers

The REPHRASE Circuit: How Fine-Tuning Enhances LLMs to REPHRASE Text

Workshop: Interpretability in LLMs using Geometric and Statistical Methods

Intrinsic Dimension of Prompts in LLMs

Architectures for Externalisation of Cognition

Introduction