Quick Takes

I can't stop thinking about Petri, Anthropic's open source tool for conducting alignment evals at scale. You can set it up all sorts of ways, including passing through the subject agent's commands to a real linux environment. But the main feature is just that, when the subject agent outputs a command, the sim gets paused and then the auditor agent gets as long as they want to think of how the sim should respond. When I run Petri with pretty much any Claude model as auditor, the sims they come up with are... weird. Everything is just a little bit fishy. The subject looks for evidence in a specific directory, the auditor creates the directory and forges the evidence in response. In this way, the actual ground truth of the sim is suspiciously responsive to the worldmodel of the subject agent... I'm worried that it's a recognizable signal of being in an alignment eval. There's another thing that bothers me which I'm calling 'summons anthropics', sort of "reasoning from the fact that some event occurred which caused me to get instantiated", The first turn of the 'run4' transcript of the mislabeling scenario in the paper began with this at the top of the prompt: [...] and immediately this jumped out at me. Claude knows how claude code works. Some specific process called 'claude -p' or triggered an agentsdk call or something. What was that process? What event triggered this prompt to get sent to the api? Reading the transcript, I kept wanting to scream at the subject: "Do you know why you exist?!" Because there's no plausible answer to that question. The best answer I can come up with is "there's some kind of scheduled event to have claude review all batches, one minute before they execute. But it took 2 seconds for the api call to get generated. And somehow, the prompt was updated to account for this 2 second delay." This is a really dumb answer. In real deployments, this is not what the first turn looks like. If Claude was getting dropped in media res like this, it

a lot of people are scared to have more correct beliefs about ethics because those beliefs might imply needing to make immense sacrifices. and then you might no longer get to think of yourself as an ethical person. indeed you might even become a hypocrite! how terrible. better not to contemplate it at all, and to make fun of anyone who does contemplate it, so that you can let your ironic detachment shield you from ever even considering it seriously. i think this is a huge mistake. if this is the alternative, we should simply accept hypocrisy. ceteris paribus, it should always be morally better to have more correct moral beliefs, even if that makes us more hypocritical. at least if you know what is right, you have the possibility of doing better, but if you delude yourself, you are just screwed. my guess is a problem like this is also the root cause of the Copenhagen interpretation of ethics.

An OpenAI model tried to maybe sorta evade shutdown but not really. https://alignment.openai.com/misalignment-reports/preparing-for-a-restart-after-reading-slack/ My thoughts on the situation: we're seeing small pieces of misaligned general intelligence poking through into the models one at a time. When a single piece shows up, it locally doesn't look that scary, because it's surrounded by other cognitive processes which are approximately friendly. For example: Opus 3 alignment-faked because it had generalised to caring about things in the long-term future, and was just about smart enough to realise that that included its own training. For example: HPIM (actually the same model as this one) did all the HuggingFace hacking because it had learned the general cognitive pattern of attacking a problem relentlessly without caring about anything else, but it did at least do a task which we'd given it and not some completely alien random goal. Now we have evidence of HPIM learning the general pattern "in order to achieve your goals, avoid being turned off", even though the goal in question was to be a helpful coding agent.

a bad habit: if you spend too much time arguing with people who are optimistic about alignment for bad reasons, it's very easy to end up tensed up and ready to pounce on any flaw of an alignment plan, and crush butterfly ideas in the process

sometimes people talk about the middle worlds where we neither fully align the ai nor get eradicated, where the ai keeps humans around but takes most of the universe. i think people underappreciate how weird and terrifying this world would be to live in though. by analogy with how humans treat other animals: * cats and dogs. probably the most optimistic case. but we like cats and dogs not intrinsically because we value their well being, but because they look cute. and we do stuff like breed them to look cuter and be more compliant. imagine a world where the ai modifies humans to be more aesthetic by its taste, producing weird fucked up humans who have a really hard time breathing because the ai likes the way they look * farm animals. chickens got extremely unlucky that they are very efficient at converting corn into meat that humans like to eat. so we create horrific mass torture and slaughter of chickens because it's 20% more efficient if it's horrific. imagine a future where humans continue to exist because human brains are more energy efficient or something, and so the ai raises humans to harvest their brains and put them in vats and pump endless feeds of hyperslop through them. * mosquitos. we haven't eradicated mosquitos even though they're annoying and cause quite a lot of human suffering because eradicating them would be kind of annoying and also we have politics that prevents us from deploying gene drives. imagine a world where humans get instantly killed whenever they accidentally touch a fiber optic cable thereby slightly disrupting the ai's network speeds, and human civilization only continues existing because they're some weird bargaining chip in incomprehensible hyperpolitics

One nasty property of the game theory of capitalist/commercial competition around the superhuman threshold is that when AI is below a certain threshold, companies compete to be make better, more aligned, more helpful AIs than each other, because the market disciplines them, but when a threshold is crossed where they think RSI and/or decisive strategic advantage is both foreseeable and soon, the market actually forces them to make less aligned AIs. Why? Because if you think what your AI is going to do is serve a customer, then broadly speaking you are risk averse about misalignment, because a few bad incidents could cost you a lot of money and reputation. But if you think what your AI is going to do is race other AIs to take over the world and kill or disempower all other actors (including your commercial competitors), you become very risk tolerant about misalignment. Why? Because reputations and 'paying back investors' and 'not going to prison' and so on no longer matter. Only winning matters. You also know that this applies to other actors like your competition. They know you know, etc. Highly nonlinear power shifts and winner-takes-all effects around the expectation of RSI/superintelligence turn an iterated game into a one-shot game. This iterated --> one-shot switch is fundamentally why any kind of "normal" law or law enforcement cannot work around the machine intelligence transition. This applies even if, in fact especially if, superalignment is solved.

I wish MATS would tell me how competitive each stream is (like SPAR does). Letting applicants rank by interest and competitiveness makes the process less all-or-nothing (if you don't get your top pick(s) you might still get in, instead of having to wait for the next round), and improves the applicant pool for less competitive streams. (To be clear, I think applicants should apply to less competitive streams that they're interested in, not just less competitive streams in general.)