Reflections on Inkhaven Would I recommend applying next year? Yes. Would I go again? It was a really good experience. I do wonder if it is the type of thing that would never be quite the same the second time. Despite thinking this, I am tempted.  Did it make me a measurably stronger writer?  I don't think so. I am much prouder of the fiction I wrote before Inkhaven than what I produced during Inkhaven. A counterpoint to this: Ozy, Wales, Linch and Eneasz convinced me I should spend some time editing/revising my fiction - which I haven’t really been doing much. This could very well make me a much stronger fiction writer, but I haven’t really tried it yet. We will see what happens with my next short story. Am I proud of what I wrote?  I am not sure. My first week, I wrote mostly terrible essays. I then started writing microfiction, and some of it still amuses me. I did get one longer bit of fiction done there, too: Lobsang’s Children. I am pretty happy with it, but it wasn’t a big hit. These are my favourite of my short posts: Harry Potter and the Rules of Quidditch Remembering Aubrey Chang With Adjustable Chef's Tool Drawer Most valuable thing about Inkhaven? Inkhaven is nothing without its people. Which residents impressed me most?  Linch struck me as the most likely to become a professional blogger.  Vishal’s blog was great and he is insanely well read. Though I didn’t really talk to him, Screwtape impressed me. He takes teaching/learning rationality seriously. By revealed preference, this isn’t something I care much about. But it is a noble pursuit and I am glad at least one person is still carrying the torch. I had a brief conversation with GeneSmith and was in awe at how many paths to intelligence augmentation he has considered, including really crazy ideas like methods for expanding adult skulls so as to have more room for brain grafts. He struck me as insanely smart.  Biggest regrets?  I was in a bad mood at the start and psyched myself out of

I find it anthropologically fascinating how at this point neurips has become mostly a summoning ritual to bring all of the ML researchers to the same city at the same time.  nobody really goes to talks anymore - even the people in the hall are often just staring at their laptops or phones. the vast majority of posters are uninteresting, and the few good ones often have a huge crowd that makes it very difficult to ask the authors questions. increasingly, the best parts of neurips are the parts outside of neurips proper. the various lunches, dinners, and parties hosted by AI companies and friend groups (and increasingly over the past few years, VCs) are core pillars of the social scene, and are where most of the socializing happens. there are so many that you can basically spend your entire neurips not going to neurips at all. at dinnertime, there are literally dozens of different events going on at the same time. multiple unofficial workshops, entirely unaffiliated with neurips, will schedule themselves to be in town at the same time; they will often have a way higher density of interesting people and ideas. if you stand around in the hallways and chat in a group long enough, eventually someone walking by will recognize someone in the group and join in, which repeats itself until the group get so big that it undergoes mitosis into smaller groups. if you're not already going to some company event, finding a restaurant at lunch or dinner time can be very challenging. every restaurant in a several mile radius will be either booked for a company event, or jam packed with people wearing neurips badges.

“If you really believed X, you would do violence about it. Therefore, not X.” I’ve seen this a few times with X := “AI extinction risk” (maybe fewer, seems like something I’d be prone to overestimate) This argument is pretty infuriating because I do really believe X, but I’m obviously not doing violence about it. So it’s transparently false (to me) and the conversation is now about whether I’m being honest about my actual beliefs, which seems to undermine the purpose of having a conversation. But it’s also kind of interesting, because it’s expressing a heuristic that seems valid - memes that incite “drastic” actions are dangerous, and activate a sort of epistemic immune system in the “uninfected.” In this particular case, it’s an immune disorder. If you absorb the whole cluster of ideas, it actually doesn’t incite particularly drastic actions in most people, at least not to the point of unilateralist violence. But the epistemic immune system attacks it immediately because it resembles something more dangerous, and it is never absorbed.    (And yes, I know that the argument is in fact invalid and there exist some X that justify violence, but I don’t think that’s really the crux) As an aside, I think a lot of “woke” ideas also undercut the basic assumptions required for a conversation to take place, which is maybe one reason the left at its ascendancy was more annoying to rationalists than the right. For instance, explaining anything you know that a woman may not know can be mansplaining, any statement from a white person can be overruled by a racial minority’s lived experience, etc., and the exchange of object-level information becomes very limited. In this case also, there is a real problem that wokeness is trying to solve, but the “solution” is too dangerous because it corrosively and steadily decays the hard-won machinery underlying productive conversation.  I think the underlying (simple) principle is that a pretty high level of basic trust is required to

Treat your obfuscated chains of thought like live bioweapons. I've spoken to a few folks at NeurIPS that are training reasoning models against monitors for various reasons (usually to figure out how to avoid unmonitorable chain of thought). I had the impression not everyone was aware how dangerous these chain of though traces were: * Make sure your obfuscated chains of thought are never used for LLM training! If obfuscated reasoning gets into the training data, this could plausibly teach models how to obfuscate their reasoning. This seems potentially pretty bad (a bit like gain of function research). I'm not saying you shouldn't do the research, it's probably worth the risk. Just make sure to keep the rollouts away from training: * Use e.g. the easy-dataset-share package (by TurnTrout et al.) to protect your dataset when you upload it somewhere (e.g. GitHub, HuggingFace). * Don't use software that trains on your files when working with dangerous material (I think the free tiers of various AI products allow for training on user data). As an example, consider the Claude 4 system card claiming that material from the Alignment Faking paper affected Claude's behaviour (discussion here). Credit to plex for bringing this issue to my attention earlier this year (with regards to my own work).

Thoughts on self-report training for honesty There's been a slew of recent papers on the idea of training models to give honest self-reports: * Li et al., Teaching LLMs to self-report their hidden objectives * Wang et al. from my group at Anthropic, studying self-report training as one of many honesty and lie detection techniques * OpenAI's Joglekar et al., Training LLMs for honesty via confessions I think that improving honesty is a really crucial goal, and I enjoyed reading (or writing) all of these papers. This quick note is a reflection on this general circle of ideas and what we have/haven't learned from these papers. All three of these papers study some variant of training models to self-report deceptive behavior. Here's what that looks like: (The above image obscures a bunch of important training details about how training is done in these papers, which I won't mention until they become relevant.) Why might self-report training work? In my view, there's two key hopes: 1. Correcting dishonest behavior. The original model would have lied, but we train it to tell the truth. 2. Inducing an honest-only output channel. This one will take a bit more explanation: 1. Consider how the <thinking> block in a reasoning model's output seems to form a special "output channel." Notably, it often doesn't seem like the model's "default assistant persona" can intentionally control the content of its thinking block. E.g. we see things like "let's hack" before reward hacking, open deceptive reasoning before covertly deceptive actions, and explicit reasoning about whether the model is undergoing evaluation. Ultimately, this is because the thinking block is subject to different training pressures, such that concealing information in the thinking block doesn't actually lead to higher reward. 2. Could we train in another special "honest-only output channel" which is never subject to training pressures that encourage dishonesty? Ideally the model would treat the

One thing I notice when reading 20th century history is that people in the 1900s-1970s had much higher priors than modern people do that the future might be radically different, in either great or terrible ways. For example: * They talked about how WW1 was the war to end all wars. They seriously talked about the prospect of banning war after WW1. Such things now sound hopelessly naive. * Serious people talked very seriously about the possibility of transformative technological change and social change following from it (e.g. Keynes/Russell speculating that people would work way fewer hours in the future). * As a minor example, between 1905-1915 Churchill spent a bunch of time trying to persuade the British government that on current trends, oil-powered ships would soon be way better than coal-powered ships, and the navy should be converted to oil power. I know of ~no recent examples where a major politician's main schtick was being thoughtful about the future of technology and making policy based on it. More generally, it was obvious after WW1 that states needed to be doing futurism and technological development in order to understand the military implications of modern technology. I really feel like the ambient cultural sense among educated Americans is: the future will be kind of like the present, treating it as if something radical will happen is naive. (They sort of say that they think climate change will be apocalyptic, but it feels to me like what they're really imagining is that the world is "enshittified" further, in the same way that it sucks that DoorDash is now expensive, and maybe poor people elsewhere die.) I think this is probably mostly because there's an important sense in which world has been changing more slowly (at least from the perspective of Americans), and the ways in which it's changing feel somehow less real. Someone who was 50 in 1945 had seen the collapse of empires that had lasted centuries, unprecedented wars, the sudden shocking r

SpaceX is amazing. As best as I (and Claude) can tell, the situation is as follows: Competent competitors to SpaceX are developing rockets that should provide around $2000/kg cost-to-orbit. This is a big improvement over legacy space competitors like ULA, Ariane, etc. which range from $5000/kg to $50,000/kg. (It's amazing that those competitors are still getting any business... the answer is nepotism/corruption basically afaict) However, these competent upcoming rockets that can do around $2000/kg? They aren't ready yet. Probably it'll be like 5 more years before they really solidly hit that milestone at scale. SpaceX, meanwhile, has already hit that milestone with Falcon 9 and has been there for years. It's why they have Starlink and nobody else does. But that's not all. SpaceX is working on Starship, which is afaict about as close to being finished as the aforementioned competitor rockets, and when it is finished it'll should provide somewhere between $15/kg and $150/kg. So... ONE TO TWO ORDERS OF MAGNITUDE CHEAPER STILL. This is a really big deal. The reason space looks the way it does today (mostly empty, a few satellites and probes, two space stations) is that it's so damn expensive to go there. SpaceX brought costs down by an OOM already and this unlocked Starlink already, and probably in the fullness of time will lead to 10x more of the other things as well (10x bigger space stations for example). Or heck, could be more than 10x; when the costs drop by 10x you often don't merely do 10x more of the same thing, you do even more than that because now a bunch of stuff is profitable that wasn't profitable before. And then once Starship is online we'll get another 1-2 OOMs on top of that. We won't have finished adjusting to the price drop from Falcon 9 (which only happened over the course of the last decade), when we'll be hit by another even bigger price drop! Intuition pump: A typical house weighs something like 50,000 kg. and costs about $300,000 to build