Quick Takes

There's a specific type of procrastination that is due to a 'fake pre-requisite' - I've started noticing these more often, and this has led to me unblocking myself a lot more. * Recently someone emailed me to ask about my postdoc plans, when I'm graduating, etc. That was a lot of stuff! I sat on it, because replying seemed to require having my whole future figured out first. So I let it go unread in my inbox. * It sat there for a while. I'd go back to it and see it often but not really know how to deal with it. It felt like it required a lot of mental energy to reply to this email. * ~2 weeks later I decided I was being really silly and I should probably just reply to this person. But doing so still felt pretty impossible - which felt really jarring. * Talking to Claude for a bit made me realize - I'd silently attached a fake prerequisite to it — "first, resolve [huge open question]" — that wasn't actually essential to the thing. * Once I reframed the ask to "what am I excited to do? and what do I currently know for sure?" - it became really easy to write the email. Effortless, in fact. And actually this was probably what was being asked in the first place So tl;dr * There is a tendency to think about vague, fuzzy tasks as monumentally large * Aversion is a pretty rational response to these tasks - it protects against burnout * Scoping down / reframing the task makes the aversion go away naturally!

From the postscript to Michael Nielsen's ASI existential risk: reconsidering alignment as a goal: [...]

i wonder how much the following hypothesis is true. it's obviously not completely true, but maybe there is some value in contemplating it. for most issues that people have expended a lot of effort arguing about, if you could truly impartially reason through it, run experiments, etc, the correct answer is either not that hard to figure out, or that we're pretty sure we can't know one way or the other. but the discourse is fucked because the vast majority (maybe literally all) of people have some bottom line they're already sympathetic to, and smart people can make plausible sounding arguments for any conclusion, and so truly reasoning things through impartially is both very hard to do and even if you somehow manage to do it, it's hard to signal that you did so, and anyways people with motivated reasoning will only listen to you if your answer happened to agree with theirs. the main evidence i have for this hypothesis is that there are questions where one side is overwhelmingly obviously correct if you actually think about it or look into it, and yet, there is the appearance of a balanced public debate. the main evidence against this hypothesis is that probably people who disagree also think they're overwhelmingly obviously correct, and it seems arrogant to declare that i am simply more correct, and in any case oftentimes people disagreeing are wrong but their disagreement still contains enough of a kernel of truth to be worth thinking about.

the nuclear option in the Senate of the US Congress, there is a "nuclear option" for overriding filibusters; a parliamentary method that can be used to ram legislation through. both sides generally agree to use it sparingly, because it's a symmetric weapon. i think there is a similar lever in arguments that is best left untouched if your goal is to actually find the truth. there is a type of argument that can be deployed in a wide range of circumstances, and is very hard to rebut except with even more nuclear arguments. the most extreme example is, suppose you are discussing the strength of some piece of evidence, and you say "well but it's impossible to truly know that, because we can't rule out the possibility that the entire external world is fake, a la Descartes's demon." or, "actually your argument is equally as conceptual as mine, because there's no such thing as empirical evidence, since all evidence must be filtered through our minds and reason applied". these kinds of arguments can be completely unbeatable and also utterly useless for answering actual questions we care about. this is closely related to the idea of an argument that proves too much, though not all arguments that prove too much are completely nuclear.

TL;DR reproducing CODI attention map interpretability results with a dashboard) CODI (Shen et al. 2025) is a latent chain-of-thought method. Like Coconut, it distills a text-based CoT into latent tokens through distillation. Unlike Coconut, which only optimizes the end-to-end model performance conditional on the CoT, CODI also aligns the hidden states of the response tokens to ensure the compressed latent CoT block is interpreted the same way as text-based CoT. The authors show performance on GSM-8k comparable to text-based CoT for GPT-2. In Section 5 of the paper, the authors of CODI show that intermediate steps in reasoning can be decoded by using the logit lens on soft tokens on GSM-8k. Additionally, as shown in the figure, CODI visualizes "attended tokens" for each latent token. These are surprisingly meaningful and show something close to the actual sequence of intermediate values in the expression tree. However, the procedure for generating these dashboards isn't described in the paper or the official codebase. The attended tokens are the top K tokens in the context by the value of the attention map, and the paper does not clarify whether this means the attention map for some head, the average of maps for all heads in some layer, or something else. I emailed author to ask about the details of the method and was told that the attention map is the average probability over all layers and heads with no other tricks. I Clauded an implementation of this aggregation and a dashboard. The intermediate results are visible in the top-1 prediction on the odd-numbered tokens, and even-numbered tokens seem to predict random characters. Recent work shows a similar pattern in Llama 3.2 1B: tokens that have an interpretable prediction under the logit lens and tokens that don't seem to correspond to anything will predictably alternate. They show that these meaningless tokens probably store the result of the previous computation, the one that was visible in logit lens on t

Here's a simple proposal for verifiable and enforceable slowdown that also differentially reallocates resources to alignment research (which I would see as one of if not the major benefits of a slowdown): Enforce that labs must spend at least X% of their compute on: 1. Giving inference capacity to external safety organizations. I think this substantially disincentivizes capabilities research, since external research won't be as relevant to the labs' specific architectures/pipelines and is more likely to leak, both effects thus making any capabilities research the lab tries to do via this mechanism much less likely to advantage them differentially (though of course they could still just end up doing bad safety research if they pick bad external safety orgs). 2. Internal safety research that the lab makes fully public. Again I think this substantially disincentivizes capabilities research, since any capabilities research that is fully public will be much less able to differentially advantage that lab specifically, as others can just copy it (though again of course the labs could still just do bad public safety research). I think this is much better than other slowdown proposals I've seen, since I think it's very concrete, verifiable, and enforceable—all the labs know exactly what they're spending their compute on, many employees know as well, and so whisteblowing is very possible—and it also differentially advantages alignment research by choosing conditions (external or fully public research) that make it much harder to do capabilities under those conditions but still pretty easy to do alignment. If you just say that compute has to go to some sort of nebulous "safety" purpose, that could easily be gamed, but I think the conditions above do a pretty good job of operationalizing criteria that are fairly hard to game. And even if you do get some gaming, on the margin you are still making capabilities research more costly, and thus buying yourself some additional sl

i think one really bad dynamic in this community is a sort of purity testing about being x risk pilled. it feels like people are constitutionally scared of considering arguments that feel like they're arguing that things might be fine in any way. tbc I've definitely been guilty of this in the past, and probably still now to some extent, but i think it's bad. maybe there is some conflict theory reason why you should orient yourself this way towards people who have an ulterior motive, but I'm like among the most x risk pilled people out there and i still find this happens when i try to discuss x risk with people.