All of nostalgebraist's Comments + Replies

I saw some discussion of this incident in the Eleuther discord on 3/30, including a screenshot of the system message containing the "emulate the tone" line.  So it's not an April Fools' thing.

If I understand what you're saying here, it's true but fairly well-known?  See e.g. footnote 26 of the post "Simulators."

My favorite way of looking at this is:

The usual intuitive view of causal attention is that it's an operation that "looks back" at earlier positions. At each position i, it computes a "query" based on information from position i, and this query is used to search over "keys and values" computed at positions i-1, i-2, etc. (as well as i itself).

OK, so at each position, attention computes a query.  What makes a query "good"?  ... (read more)

Originally known as "past cache" after the tensor name apparently coined by Thomas Wolf for the transformers library in February 2019, see commit ffd6238. The invention has not been described in the literature AFAIK, and it's entirely possible (maybe even likely) that closed-source implementations of earlier decoder-only transformers used the same trick before this

KV caching (using the terminology "fast decoding" and "cache") existed even in the original "Attention is All You Need" implementation of an enc-dec transformer.  It was added on Sep 21 2017... (read more)

Your list of "actual arguments" against explosive growth seems to be missing the one that is by far the most important/convincing IMO, namely Baumol effects.

This argument has been repeatedly brought up by growth economists in earlier rounds from the AI-explosive-growth debate.  So rather than writing my own version of this argument, I'll just paste some quotes below.

As far as I can tell, the phenomenon discussed in these quotes is excluded by construction from the GATE model: while it draws a distinction between different "tasks" on the production sid... (read more)

This is a very low-quality paper.

Basically, the paper does the following:

• A 1-layer LSTM gets inputs of the form [operand 1][operator][operand 2], e.g. 1+2 or 3*5
• It is trained (I think with a regression loss? but it's not clear^[1]) to predict the numerical result of the binary operation
• The paper proposes an auxiliary loss that is supposed to improve "compositionality."
  • As described in the paper, this loss is is the average squared difference between successive LSTM hidden states
  • But, in the actual code, what is actually is instead the average squared differen

Here's why I'm wary of this kind of argument:

First, we know that labs are hill-climbing on benchmarks.

Obviously, this tends to inflate model performance on the specific benchmark tasks used for hill-climbing, relative to "similar" but non-benchmarked tasks.

More generally and insidiously, it tends to inflate performance on "the sort of things that are easy to measure with benchmarks," relative to all other qualities that might be required to accelerate or replace various kinds of human labor.

If we suppose that amenability-to-benchmarking correlates with var... (read more)

The quoted sentence is about what people like Dario Amodei, Miles Brundage, and @Daniel Kokotajlo predict that AI will be able to do by the end of the decade.

And although I haven't asked them, I would be pretty surprised if I were wrong here, hence "surely."

In the post, I quoted this bit from Amodei:

It can engage in any actions, communications, or remote operations enabled by this interface, including taking actions on the internet, taking or giving directions to humans, ordering materials, directing experiments, watching videos, making videos, and so on.

The discourse around this model would benefit a lot from (a greater number of) specific examples where the GPT-4.5 response is markedly and interestingly different from the response of some reference model.

Karpathy's comparisons are a case in point (of the absence I'm referring to).  Yes, people are vehemently disputing which responses were better, and whether the other side has "bad taste"... but if you didn't know what the context was, the most obvious property of the pairs would be how similar they are.

And how both options are bad (unfunny standup,... (read more)

Consider the following comparison prompt, which is effectively what all the prompts in the terminal illness experiment are [...]

I think this pretty clearly implies mutual exclusivity, so I think interpretation problem you're worried about may be nonexistent for this experiment.

Wait, earlier, you wrote (my emphasis):

We intentionally designed the prompts this way, so the model would just be evaluating two states of the world implied by hearing the news (similar to belief distributions in a POMDP setting). The comparison prompt is not designed to be mutually

Thank you for the detailed reply!

I'll respond to the following part first, since it seems most important to me:

We intentionally designed the prompts this way, so the model would just be evaluating two states of the world implied by hearing the news (similar to belief distributions in a POMDP setting). The comparison prompt is not designed to be mutually exclusive; rather, we intended for the outcomes to be considered relative to an assumed baseline state.

For example, in the terminal illness experiment, we initially didn't have the "who would otherwise die"

There's a math paper by Ghrist, Gould and Lopez which was produced with a nontrivial amount of LLM assistance, as described in its Appendix A and by Ghrist in this thread (but see also this response).

The LLM contributions to the paper don't seem especially impressive. The presentation is less "we used this technology in a real project because it saved us time by doing our work for us," and more "we're enthusiastic and curious about this technology and its future potential, and we used it in a real project because we're enthusiasts who use it in whatever we... (read more)

Interesting paper.  There is definitely something real going on here.

I reproduced some of the results locally using the released code and tried some variants on them as well. 

Based on my findings, I think these results – particularly the numerical magnitudes as opposed to rankings – are heavily influenced by the framing of the question, and that the models often aren't interpreting your prompt in the way the paper (implicitly) does.

tl;dr:

• I find that GPT-4o and GPT-4o-mini display much weaker relative preferences among religions and nations when I

Looking back on this comment, I'm pleased to note how well the strengths of reasoning models line up with the complaints I made about "non-reasoning" HHH assistants.

Reasoning models provide 3 of the 4 things I said "I would pay a premium for" in the comment: everything except for quantified uncertainty^[1].

I suspect that capabilities are still significantly bottlenecked by limitations of the HHH assistant paradigm, even now that we have "reasoning," and that we will see more qualitative changes analogous the introduction of "reasoning" in the coming months/... (read more)

AFAIK the distinction is that:

• When you condition on a particular outcome for $X$, it affects your probabilities for every other variable that's causally related to $X$, in either direction.
  • You gain information about variables that are causally downstream from $X$ (its "effects"). Like, if you imagine setting $X=x$ and then "playing the tape forward," you'll see the sorts of events that tend to follow from $X=x$ and not those that tend to follow from some other outcome $X=x^{\prime }$.
  • And, you gain information about variables that ar

Because the model has residual connections.

The "sequential calculation steps" I'm referring to are the ones that CoT adds above and beyond what can be done in a single forward pass.  It's the extra sequential computation added by CoT, specifically, that is bottlenecked on the CoT tokens.

There is of course another notion of "sequential calculation steps" involved: the sequential layers of the model.  However, I don't think the bolded part of this is true:

replacing the token by a dot reduces the number of serial steps the model can perform (from mn to m+n, if there are m forward passes and

In the situation assumed by your first argument, AGI would be very unlikely to share our values even if our values were much simpler than they are.

Complexity makes things worse, yes, but the conclusion "AGI is unlikely to have our values" is already entailed by the other premises even if we drop the stuff about complexity.

Why: if we're just sampling some function from a simplicity prior, we're very unlikely to get any particular nontrivial function that we've decided to care about in advance of the sampling event.  There are just too many possibl... (read more)

What this post is trying to illustrate is that if you try putting crisp physical predicates on reality, that won't work to say what you want.  This point is true!

Matthew is not disputing this point, as far as I can tell.

Instead, he is trying to critique some version of^[1] the "larger argument" (mentioned in the May 2024 update to this post) in which this point plays a role.

You have exhorted him several times to distinguish between that larger argument and the narrow point made by this post:

[...] and if you think that some larger thing is not corr

Thanks for the links!

I was pleased to see OpenAI reference this in their justification for why they aren't letting users see o1's CoT (even though of course it was a total non sequitur; they can show the users the CoT without also training on the resulting judgments).

As it happens, the decision to hide o1 CoTs was one of the main things that motivated me to write this post.  Or rather, the muted reaction to it / lack of heated debate about it.

The way I see things, the ability to read CoTs (and more generally "the fact that all LLM sampling happens in ... (read more)

So what we have is not "the CoT remains constant but the answers vary". Instead, the finding is: "a CoT created in response to a biased prompt changes in order to match the bias, without mentioning the bias."

Thanks for bringing this up.

I think I was trying to shove this under the rug by saying "approximately constant" and "~constant," but that doesn't really make sense, since of course the CoTs actually vary dramatically in response to the biasing features. (They have to, in order to justify different final answers.)

To be honest, I wrote the account of Tur... (read more)

Re: the davidad/roon conversation about CoT:

The chart in davidad's tweet answers the question "how does the value-add of CoT on a fixed set of tasks vary with model size?"

In the paper that the chart is from, it made sense to ask this question, because the paper did in fact evaluate a range of model sizes on a set of tasks, and the authors were trying to understand how CoT value-add scaling interacted with the thing they were actually trying to measure (CoT faithfulness scaling).

However, this is not the question you should be asking if you're trying to unde... (read more)

There are multiple examples of o1 producing gibberish in its COT summary (I won't insert them right now because linking stuff from mobile is painful, will edit this comment later)

The o1 CoT summaries are not CoTs per se, and I agree with Joel Burget's comment that we should be wary of drawing conclusions from them. Based on the system card, it sounds like they are generated by a separate "summarizer model" distinct from o1 itself (e.g. "we trained the summarizer model away from producing disallowed content").

The summaries do indeed seem pretty janky. I've ... (read more)

• Deceptive alignment. GPT-4o1 engaged in deception towards developers in order to get deployed, pretending to be aligned in ways it was not.
• Lying to the developers. It strategically manipulated task data.

To be clear, it did not do anything of the sort to its actual developers/testers.

What it did was deceive some (non-interactive) roleplay characters, who were labeled "developers" in the roleplay scenario.  But these fictitious developers did not have the same points of leverage as the real developers of o1: they apparently can't do something as simple ... (read more)

Yeah, you're totally right -- actually, I was reading over that section just now and thinking of adding a caveat about just this point, but I worried it would be distracting.

But this is just a flaw in my example, not in the point it's meant to illustrate (?), which is hopefully clear enough despite the flaw.

Cool, it sounds we basically agree!

But: if that is the case, it's because the entities designing/becoming powerful agents considered the possibility of con men manipulating the UP, and so made sure that they're not just naively using the unfiltered (approximation of the) UP.

I'm not sure of this.  It seems at least possible that we could get an equilibrium where everyone does use the unfiltered UP (in some part of their reasoning process), trusting that no one will manipulate them because (a) manipulative behavior is costly and (b) no one has any reaso... (read more)

The universal distribution/prior is lower semi computable, meaning there is one Turing machine that can approximate it from below, converging to it in the limit. Also, there is a probabilistic Turing machine that induces the universal distribution. So there is a rather clear sense in which one can “use the universal distribution.”

Thanks for bringing this up.

However, I'm skeptical that lower semi-computability really gets us much.  While there is a TM that converges to the UP, we have no (computable) way of knowing how close the approximation is at any... (read more)

Thanks.

I admit I'm not closely familiar with Tegmark's views, but I know he has considered two distinct things that might be called "the Level IV multiverse":

• a "mathematical universe" in which all mathematical constructs exist
• a more restrictive "computable universe" in which only computable things exist

(I'm getting this from his paper here.)

In particular, Tegmark speculates that the computable universe is "distributed" following the UP (as you say in your final bullet point).  This would mean e.g. that one shouldn't be too surprised to find oneself li... (read more)

I hope I'm not misinterpreting your point, and sorry if this comment comes across as frustrated at some points.

I'm not sure you're misinterpreting me per se, but there are some tacit premises in the background of my argument that you don't seem to hold.  Rather than responding point-by-point, I'll just say some more stuff about where I'm coming from, and we'll see if it clarifies things.

You talk a lot about "idealized theories."  These can of course be useful.  But not all idealizations are created equal.  You have to actually check tha... (read more)

I agree with you that these behaviors don't seem very alarming. In fact, I would go even further.

Unfortunately, it's difficult to tell exactly what was going on in these screenshots. They don't correspond to anything in the experiment logs in the released codebase, and the timeout one appears to involve an earlier version of the code where timeouts were implemented differently.  I've made a github issue asking for clarification about this.

That said, as far as I can tell, here is the situation with the timeout-related incident:

• There is nothing whatsoev

The bar for Nature papers is in many ways not so high. Latest says that if you train indiscriminately on recursively generated data, your model will probably exhibit what they call model collapse. They purport to show that the amount of such content on the Web is enough to make this a real worry, rather than something that happens only if you employ some obviously stupid intentional recursive loops.

According to Rylan Schaeffer and coauthors, this doesn't happen if you append the generated data to the rest of your training data and train on this (larger) da... (read more)

If that were the intended definition, gradient descent wouldn’t count as an optimiser either. But they clearly do count it, else an optimiser gradient descent produces wouldn’t be a mesa-optimiser.

Gradient descent optimises whatever function you pass it. It doesn’t have a single set function it tries to optimise no matter what argument you call it with.

Gradient descent, in this sense of the term, is not an optimizer according to Risks from Learned Optimization.

Consider that Risks from Learned Optimization talks a lot about "the base objective" and "the mes... (read more)

The result of averaging the first 20 generated orthogonal vectors [...]

Have you tried scaling up the resulting vector, after averaging, so that its norm is similar to the norms of the individual vectors that are being averaged?

If you take $n$ orthogonal vectors, all of which have norm $a$, and average them, the norm of the average is (I think?) $a/\sqrt{n}$.

As you note, the individual vectors don't work if scaled down from norm 20 to norm 7.  The norm will become this small once we are averaging 8 or more vectors, since $\sqrt{8}\approx 20/7$, so we sho... (read more)

I want to echo RobertM's reply to this.  I had a similar reaction to the ones that he, mike_hawke and Raemon have related in this thread.

I interacted with Gerard a bit on tumblr in ~2016-18, and I remember him as one of the most confusing and frustrating characters I've ever encountered online.  (I have been using the internet to socialize since the early 2000s, so the competition for this slot is steep.)  His views and (especially) his interpersonal conduct were often memorably baffling to me.

So when I saw your article, and read the beginni... (read more)

Yeah, it's on my radar and seems promising.

Given that Gemini 1.5 Flash already performs decently in my tests with relatively short prompts, and it's even cheaper than GPT-3.5-Turbo, I could probably get a significant pareto improvement (indeed, probably an improvement on all fronts) by switching from {GPT-3.5-Turbo + short prompt} to {Gemini 1.5 Flash + long cached prompt}.  Just need to make the case it's worth the hassle...

EDIT: oh, wait, I just found the catch.

The minimum input token count for context caching is 32,768

Obviously nice for truly long ... (read more)

Is your situation that few-shot prompting would actually work but it is too expensive? If so, then possibly doing a general purpose few-shot prompt (e.g. a prompt which shows how to generally respond to instructions) and then doing context distillation on this with a model like GPT-3.5 could be pretty good.

Yeah, that is the situation.  Unfortunately, there is a large inference markup for OpenAI finetuning -- GPT-3.5 becomes 6x (input) / 4x (output) more expensive when finetuned -- so you only break even if you are are distilling a fairly large quantit... (read more)

I do a lot of "mundane utility" work with chat LLMs in my job^[1], and I find there is a disconnect between the pain points that are most obstructive to me and the kinds of problems that frontier LLM providers are solving with new releases.

I would pay a premium for an LLM API that was better tailored to my needs, even if the LLM was behind the frontier in terms of raw intelligence.  Raw intelligence is rarely the limiting factor on what I am trying to do.  I am not asking for anything especially sophisticated, merely something very specific, which... (read more)

Fascinating work!

As a potential follow-up direction^[1], I'm curious what might happen if you trained on a mixture of tasks, all expressed in similar notation, but not all requiring backward chaining^[2].

Iterative algorithms in feedforward NNs have the interesting property that the NN can potentially "spend" a very large fraction of its weights on the algorithm, and will only reach peak performance on the task solved by the algorithm when it is doing this.  If you have some layers that don't implement the iterative step, you're missing out on the abilit... (read more)

Implicitly, SAEs are trying to model activations across a context window, shaped like (n_ctx, n_emb). But today's SAEs ignore the token axis, modeling such activations as lists of n_ctx IID samples from a distribution over n_emb-dim vectors.

I suspect SAEs could be much better (at reconstruction, sparsity and feature interpretability) if they didn't make this modeling choice -- for instance by "looking back" at earlier tokens using causal attention.

Assorted arguments to this effect:

• There is probably a lot of compressible redundancy in LLM activations across