Replying toWill Jesus Christ return in an election year?

Will Jesus Christ return in an election year?

The Incorrect Resolution hypothesis rings false because, while there's some chance of an incorrect resolution, it's really unlikely to be as high as 3%. Polymarket has a lot of reputation to lose by incorrectly resolving this market, and would almost certainly override its consensus-based resolution mechanism if it came to that.

I think historically UMA oracle token holder profiteers have misresolved some markets without Polymarket overriding.

I would guess profiteers would focus on markets with more liquidity. I would guess that the base rate so far has been much less than 3%, but maybe there are lots of resolved markets with lower trading volume ('Will Jesus Christ...' has $400K in volume). It does seem like it would be unusually bad for Polymarket if this particular market was misresolved.

It seems not impossible to me for many markets to be misresolved by UMA in one big attack, but I don't know how UMA and Polymarket work.

Noa Nabeshima9moQuick Take

While language models plausibly are trained with comparable amounts of FLOP to humans today here are some differences:

Humans process much less data
Humans spend much more compute per datapoint
Human data includes them taking actions and the results of those actions, language model pretraining data much less so.

These might explain some of the strengths/weaknesses of language models

LMs know many more things than humans, but often in shallower ways.
LMs seem less sample-efficient than humans (less compute per datapoint and they haven't been very optimized for sample-efficiency yet)
LMs are worse at taking actions over time than humans.

Replying toEfficient Dictionary Learning with Switch Sparse Autoencoders

Noa Nabeshima11mo*

Efficient Dictionary Learning with Switch Sparse Autoencoders

If this is your implementation:

https://github.com/amudide/switch_sae/blob/main/dictionary_learning/trainers/switch.py

It looks like you when you encode you do a dense encoder forward pass and then mask using the expert router.

I think this means that the FLOP scaling laws claim is misleading because (my impression is that) your current train code uses much more FLOP than the scaling law graphs, because it calculates every expert's activations for every input.

But I think the empirical claims about the learned features and the FLOP scaling laws still should hold up for implementations that actually do the conditional computations.

I also expect H100/B100-time scaling charts than FLOP charts to be more informative for future work because I now think memory-bandwidth has decent odds of being the main bottleneck for training time.

Noa Nabeshima1yQuick Take

One barrier to SAE circuits is that it's currently hard to understand how attention out SAE latents are calculated. Even if you do IG attribution patching to try to understand which earlier latents are relevant to the attention out SAE latents, it doesn't tell you how these latents interact inside the attention layer at all.

Noa Nabeshima1yQuick Take

Auto-interp is currently really really bad

I think o1 is the only model that seems to perform decently at auto-interp but it's very expensive! IE $1/latent label. This is frustrating to me.

Noa Nabeshima1y*Quick Take

TinyModel SAEs have these first entity and second entity latents.

E.g. if the story is 'Once upon a time Tim met his friend Sally.', Tim is the first entity and Sally is the second entity. The latents fire on all instances of first|second entity after the first introduction of that entity.

I think I at one point found an 'object owned by second entity' latent but have had trouble finding it again.

I wonder if LMs are generating these reusable 'pointers' and then doing computation with the pointers. For example to track that an object is owned by the first entity, you just need to calculate which entities are instances of the first entity, calculate... (read more)

Noa Nabeshima's Shortform

Noa Nabeshima

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Replying toMatryoshka Sparse Autoencoders

Noa Nabeshima1y

Matryoshka Sparse Autoencoders

You know I was thinking ab this-- say that there are two children and they're orthogonal to the parent and each have probability 0.4 given the parent. If you imagine the space it looks like three clusters, two with probability 0.4, norm 1.4 and one with probability 0.2 and norm 1. They all have high cosine similarity with each other. From this frame, having the parent 'include' the children directions a bit doesn't seem that inappropriate. One SAE latent setup that seems pretty reasonable is to actually have one parent latent that's like "one of these three clusters is active" and three child latents pointing to each of the three clusters. The parent latent decoder in that setup would also include a bit of the child feature directions.

This is all sketchy though. It doesn't feel like we have a good answer to the question "How exactly do we want the SAEs to behave in various scenarios?"

Replying toMatryoshka Sparse Autoencoders

Noa Nabeshima1y

Matryoshka Sparse Autoencoders

This is cool! I wonder if it can be fixed. I imagine it could be improved some amount by nudging the prefix distribution, but it doesn't seem like that will solve it properly. Curious if this is a large issue in real LMs. It's frustrating that there aren't ground-truth features we have access to in language models.

I think how large of a problem this is can probably be inferred from a description of the feature distribution. It'd be nice to have a better sense of what that distribution is (assuming the paradigm is correct enough).

Replying toMatryoshka Sparse Autoencoders

Noa Nabeshima1y*

Matryoshka Sparse Autoencoders

Tree Methodology

To generate the trees in sparselatents.com/tree_view, I use a variant of Masked Cosine Similarity (MCS), a metric introduced in Towards Monosemanticity. The original MCS is calculated like this: For any two latents A and B, first compute the cosine similarity between their activations, but only considering tokens where latent A is active. Then compute the same similarity, but only for tokens where latent B is active. The final MCS value is the larger of these two similarities.

Instead of taking the max, I do a directed MCS where I just consider the cosine similarity between A and B's activations on tokens where B is active. Then, I multiply this directed MCS score... (read more)

Replying toMatryoshka Sparse Autoencoders

Noa Nabeshima1y

Matryoshka Sparse Autoencoders

The Matryoshka SAE trained on the toy model learn the true features on most runs, not all of them. Sometimes a small number of latents, modally one, seem to get stuck in a bad state.

Matryoshka Sparse Autoencoders

Noa Nabeshima

View trees here
Search through latents with a token-regex language
View individual latents here
See code here (github.com/noanabeshima/matryoshka-saes)
Alternate version of this document with appropriate-height interactives.

Abstract

Sparse autoencoders (SAEs)^[1]^[2] break down neural network internals into components called latents. Smaller SAE latents seem to correspond to more abstract concepts while larger SAE latents seem to represent finer, more specific concepts.

While increasing SAE size allows for finer-grained representations, it also introduces two key problems: feature absorption introduced in Chanin et al. ^[3], where latents develop unintuitive "holes" as other latents in the SAE take over specific cases, and what I term fragmentation, where meaningful abstract concepts in the small SAE (e.g. 'female names' or 'words in quotes') shatter (via... (read 3078 more words →)

Poll: Which variables are most strategically relevant?

Daniel Kokotajlo

Daniel Kokotajlo, Noa Nabeshima, Noa Nabeshima

Which variables are most important for predicting and influencing how AI goes?

Here are some examples:

Timelines: “When will crazy AI stuff start to happen?”
Alignment tax: “How much more difficult will it be to create an aligned AI vs an unaligned AI when it becomes possible to create powerful AI?
Homogeneity: "Will transformative AI systems be trained/created all in the same way?"
Unipolar / Multipolar: "Will transformative AI systems be controlled by one organization or many?"
Takeoff speeds: "Will takeoff be fast or slow (or hard or soft, etc.)?"

We made this question to crowd-source more entries for our list, along with operationalizations and judgments of relative importance. This is the first step of a... (read 155 more words →)

LESSWRONG
LW

LESSWRONG
LW

Noa Nabeshima

Noa Nabeshima

Noa Nabeshima

Noa Nabeshima's Shortform

Matryoshka Sparse Autoencoders

Poll: Which variables are most strategically relevant?

Noa Nabeshima

Noa Nabeshima

Noa Nabeshima

Noa Nabeshima's Shortform

Matryoshka Sparse Autoencoders

Poll: Which variables are most strategically relevant?

Tree Methodology

Abstract