Quick Takes

You suspect someone in your community is a bad actor. Kinds of reasons not to move against them: 1. You're uncertain 1. Especially if your uncertainty will likely be largely resolved soon 2. You lack legible evidence (or other ways of convincing others), and they're not already seen as sketchy 1. Especially if you'll likely get better legible evidence soon 3. They're doing some good stuff; you need them for some good stuff 4. They're popular, politically powerful, or have power to hurt you 1. And so you'd fail 1. And maybe you'd get kicked out or lose power (especially if your move is unpopular or considered inappropriate, or "community" is more like "team" or "circle") 2. And so making them an enemy of [you or your community] is costly 5. Personal psychological costs 6. It's just time-consuming to start a fight, especially because they'll be invested in discrediting you and your claims This is all from-first-principles. I'm interested in takes and reading recommendations. Also creative affordances / social technology. The convenient case would be that you can privately get the powerful stakeholders on board and then they all oust the bad actor and it's a fait accompli, so there's no protracted conflict and minimal cost to you and your community. If you can't get the powerful stakeholders on board, I guess you give up and just privately share information with people you trust — unless you're sufficiently powerful/independent/ostracized that it's cheap for you to make enemies, or it's sufficiently important, or you can share information anonymously. If you're scared about telling the powerful stakeholders, I guess it's the same situation. This is assuming that the main audience for your beliefs is some powerful stakeholders. Sometimes it's more like many community members.

I really liked Scott Alexander's post on AI consciousness. He doesn't really discuss the paper of Butlin et al which is fine by me. I can never get excited about consciousness research. It always seemed like drawing the target around the dart, where the dart is humans, and you're trying (not always successfully) to make a definition that includes all humans but doesn't include thermostats. I can't get that excited about whether or not LLM have the "something something feedback" or whether having chain of thoughts changes anything. Maybe, like Scott says, it's like Aphantasia and I'm just not conscious in the same way that some other people are. (Do you have to be conscious to write consciousness research? This seems like another one of these hard questions that can't be answered based on observed output alone.) In any case, the main thing people care about is not consciousness but moral patienthood. And here I agree with Scott that, regardless of what philosphers might wish was the case, in practice it's not really about internal or intelligence but about form factor and appearance, together with our selfish interest. If AI's are embodied, look like us or are super cute it will be hard not to ascribe to them consciousness and some amount of moral patienthood. And arguably rightly so. You don't want to teach your kids that it's OK to mistreat things that look very much like humans.  If AI's are served in the form of faceless assistants, then we would likely not care. If anything, the more super-intelligent AI is, the more alien it would seem to us, and so the less we would care about its welfare. Some of it might also be how useful or painful for us to ascribe moral patienthood to the AIs.  As Scott says, we might have ascribed some degree of personhood to pigs if they didn't taste so good.

Recently I was dragged on Twitter about my timelines & AI 2027; here is my main response: https://x.com/DKokotajlo/status/1992316608073847201 

Back in 2020, @Raemon gave me some extremely good advice. @johnswentworth had left some comments on a post of mine that I found extremely frustrating and counterproductive. At the time I had no idea about his body of work, so he was just some annoying guy. Ray, who did know who John was and thought he was doing important work, told me: Which didn't mean I had to heal the rift with John in particular, but if I was going to make that a policy then I would need to give up on my goal of having real impact.  John and I did a video call, and it went well. He pointed out a major flaw in my post, I impressed him by immediately updating once he pointed it out. I still think his original comments displayed status dynamics while sneering at them, and find that frustrating, but Ray was right that not all factual corrections will be delivered in pleasing forms. 

We have finally solved an age old problem in philosophy: * Gemini 3 pro is 1.2 cents per thousand tokens. * Gemini 3 pro image is 13.4 cents per image. Therefore an image is worth 11167 words, not 1000 as the classicists would have it.

Inoculation Prompting has to be one of the most janky ad-hoc alignment solutions I've ever seen. I agree that it seems to work for existing models, but I expect it to fail for more capable models in a generation or two. One way this could happen: 1) We train a model using inoculation prompting, with a lot of RL, using say 10x the compute for RL as used in pretraining 2) The model develops strong drives towards e.g. reward hacking, deception, power-seeking because this is rewarded in the training environment 3) In the production environment, we remove the statement saying that reward hacking is okay, and replace it perhaps with a statement politely asking the model not to reward hack/be misaligned (or nothing at all) 4) The model reflects upon this statement ... and is broadly misaligned anyway, because of the habits/drives developed in step 2. Perhaps it reveals this only rarely when it's confident it won't be caught and modified as a result.  My guess is that the current models don't generalize this way because the amount of optimization pressure applied during RL is small relative to e.g. the HHH prior. I'd be interested to see a scaling analysis of this question. 

It's common knowledge on lesswrong that "0 and 1 are not probabilities". This means that if you update using Bayes law,  your prior is 0 or 1 iff your posterior is 0 or 1. In other words the only way to be absolutely certain, is if you already were certain before considering any evidence. You obviously shouldn't be certain/unconvincable without evidence so you shouldn't ever be. But only this week, I thought of an important corollary:  precise measurement is impossible.  If X is a continuous random variable, then P(X=x)=0 for any x. Because 0 and 1 are not probabilities, this means you'll never know the precise value of X. You can never measure something continuous precisely. I think it's suspicious to get this conclusion from an "armchair" argument. I've deduced something important about the world without taking into account reality, without evidence. Which I said you shouldn't ever do! So where did I go wrong?