Quick Takes

FWIW Alex Bores seems like a very mildly below-average integrity politician, having talked to him once and having followed his campaign and social media presence. He seems to say things he doesn’t believe somewhat more often than other politicians, but not much so, and he gives me some amount of "naive-consequentialist EA" vibes that make me think he is higher variance on this dimension than others. He does seem to really care about the AI Safety thing, he really appears to be targeted by a ton of very aggressive attack ads funded by AI capability companies, and it seems really pretty important to counteract that, all of which make me think it's overall good to vote for him (and to do high-integrity campaigning for him). His behavior on social media and speeches is very far below the integrity standards for leaders in the AI safety community, and if he behaved similarly here as he behaves in politics, I would advocate strongly against empowering him. Of course politics is a different playing field, and as I said, he seems roughly average for that domain. But if, as a result of people in the AI Safety community writing glorious exhortations of him, you ended up with the impression that when Alex says something you can take it remotely as seriously as posts around here, please update. Most of his social media posts and campaign actions appear to be things he doesn’t believe and is choosing to say for political reasons, including on AI topics (on which he happily conflates existential risks with things like trying to blame AI systems for triggering suicides in teenagers). I tentatively think this is OK given the standards in politics, and you ultimately have to vote for someone, but it’s also not great. IMO advocating for regulating AI on the basis of spurious harms like this is bad, will eventually backfire, and should be modeled as a cost, not as a benefit, from the perspective of predicting how useful his actions will be on AI safety topics. If you heard about him

I have strong mixed feelings about the current state of labs' alignment science blogs. * On one hand, I think it's good they are making an effort to communicate what they are doing. * On the other hand, I think the current level of communication is far below what I'd like it to be. To be clear, my bar for scientific publications is that they should enable an external party to do a reproduction. I have been disappointed by both Anthropic and OpenAI in this regard. A couple recent examples: * Anthropic's "Teaching claude why" is incredibly scrubbed of relevant scientific detail, to the point that graphs don't have meaningful labels * OpenAI's "Beneficial RL" is devoid of any meaningful detail about the titular method, to the point that it's impossible to understand what they did or how to reproduce this. To be clear, I appreciate that these were published at all. But IMO it's hard to view these examples as much more than safety-washing. Side note: I want to register appreciation for GDM's alignment team which has been putting out research notes that are much more detailed than the above two examples, e.g. this post on SDF which IMO just clearly is a better scientific publication than "teaching claude why" Also - this wasn't always the case! Just a couple years ago we had "Constitutional AI" and "Deliberative Alignment" which were extremely detailed technical accounts of their leading alignment procedures.

do you have an ambitious idea for how to make AGI go well? do you need money? do you hate bureaucracy and friction? apply now for microgrants! please read the entire doc before applying. only send applications to the designated location, or they will be automatically rejected. https://docs.google.com/document/d/10zAp2bXTkZgiPreIm4crp38TFco4KleFN14Kw5BprAs/edit?tab=t.0

The AI safety ecosystem is so well resourced that it has been correctly identified by many as one of the best paths into high prestige AI research jobs. This person on twitter has written a popular article about getting into frontier ai labs and a Field Guide to AI Fellowships. The "AI Fellowships" are mostly AI safety programs funded by CG/OpenPhil. I have also noticed that people in ML research are quite likely to have heard of MATS and be interested in participating, even when they have very little interest in AI safety. Idk how good or bad this is. It definitely causes a lot of ML researchers to engage with the AI safety literature, where they otherwise would not have. But it's worth noting that while the primary driver of applications to programs like MATS used be concern about AI safety, now it is increasingly a desire to work at a frontier lab. I used to select participants for LASR labs, one of programs listed, and we actively tried to choose people who cared about AI safety, but I think we often did not succeed and indeed some people on the program now work at frontier labs in roles that I think have little to do with safety. Safety-washing in practice mostly looks like people rationalizing research or jobs that they would have done anyway as necessary for safety. It's easy to do because it is in fact genuinely unclear in most cases what is helpful and harmful for safety. Only by looking at the larger pattern can we notice the suspicious abundance of conveniently overlapping opportunities that further both safety and a person's short-term interests. However, I think it will be less common in people whose primary motivation is to prevent AI catastrophe.

notes! I've just posted a repo with a bunch of my notes from 2023–2026, mostly on topics with relevance to AI alignment; see here for more meta information. An assortment of 101 items from the vault: * AI safety presentation slides (or really self-contained slideuments): * Variants of the alignment problem (at the AFFINE Superintelligence Alignment Seminar) * Verification-based alignment schemes (at the AFFINE Superintelligence Alignment Seminar) * Inspection of thinking as a source of ideas for understanding minds (at the AFFINE Superintelligence Alignment Seminar) * Model-wise thinking (at the AFFINE Superintelligence Alignment Seminar) * Intro to AGI safety (at the University of Tartu) * Impact cases for guardrails/monitoring/verification (at Mila) * Why fear a math AI? (at Mila) * on understanding: * good concepts are concepts that support inferences * predicting specific things * model-thinking in mathematics * how categoricity (or universal properties) relate to the implementation of mental structures * gaining the integral notion * cheap inferences * turning activity-situations into abstract games * a stylized genealogy of sonic concepts * condensation: * condensation coefficients from scale-freeness * condensation coefficients compatible with the loss being a positive combination of subset losses * linear condensation * limitations of condensation * condensation coding * valuing, ethics, metaethics: * mundane malignity * it is natural for a mind to remake the world * terminality causes unmooredness * seeing (meta)ethics clearer via mathematical logic * centering ethics on coming up with ways to act * there are many good and important value-flavored structures * the epistemics-ethics bridge * in one sense, human values are very complicated; in another sense, human values are very simple * argument against universal value extraction device * an illustrat

If Eliezer every writes a memoir, it should be structured as a time loop novel. Loop 1: e/acc Eliezer races to defeat death by forming a coalition to build AI as fast as possible. AI kills everyone. Somehow (mumble mumble acausal trade simulation mumble) he finds himself in back at the start with another chance. Loop 2: Eliezer realizes he needs to solve alignment first, spends a loop working on this, then someone else builds AI and everyone dies. Loop 3: Eliezer loses hope, decides to just write fanfics. Accidentally realizes that if you structure a textbook as fanfic people will actually read it. Eventually everyone dies. Loop 4: Our timeline, Eliezer realizes something about the time loop is destabilizing the timeline. Russia is aggressively starting fights with Ukraine and the EU, risking nuclear war, China is threatening its neighbors, etc. Realizing this could be the final loop before things truly go crazy, he goes all out... Readers, vote for your ending: (1) Convince governments to ban AI, (2) Convince AI companies not to build AI, (3) Make AI solve the alignment problem, (4) YOLO, maybe it'll just work out this time. Side plot: Bringing famous social network influencer Elon Musk into the time loop so he can draw attention to the problem, which unfortunately backfires.

would people find an alignment microgrant program useful? handing out grants on the order of $10k with very minimal bureaucracy - a 15 minute application process and a 30 minute retrospective call 3 months down the line. is this a quantity of money that enough to matter at all? is overhead around grant applications actually a big issue? is there really good work that nobody is funding already?