Quick Takes

I donated to Pause AI UK today. Why this org in particular? 1) Right now, AI advocacy is clearly the most important thing to do. 2) I've been to some of their protest and one more event, so I know they are doing good work. I think that donating "close to home" in the sense of point 2, is underrated in EA.

Might we think (not merely reason) less and less as reasoning, feeling, and acting take up more of our time? I recently finished a reading group, AI Safety × Philosophy Reading Group 1, focused on broad philosophical questions about this new technology and even received a fancy certificate 2 (hooray). Here is what the group gave me and what I took away. The main things this course provided were a structure and a reading list. Each week, the structure comprised (1) a meeting with (2) a detailed minute-by-minute agenda with concrete questions to surface participants' positions and to facilitate a debate during the meeting, and (3) a reading list with a core question. The reading list pointed out key pieces that I'd have been unlikely to spend time on alone, given the sheer volume of information, e.g. Thomas Nagel's ‘What Is It Like to Be a Bat?’ (1974), which seemed too old to me but turned out to be insightful, as I wrote before 3. Another structural thing was that this course was in English and I was eager to practice discussing these philosophical questions in that language because language shapes how we think. The key thing I took from this course is that being, or what exists, connects such things as value alignment, consciousness, understanding, rights, control (delegation), and simulation (the experience machine) as applied to AI. For example, value alignment depends on what people value or not, which in turn depends on what people believe exists or cannot exist (no one values what they believe cannot exist), and the question of consciousness requires the answer to the question of whether there is something it is like to be someone else (another human being, animal, artificial mind) (Nagel). Therefore value alignment and consciousness are connected to the question of what being is. It took millennia for people to understand and write down those ideas, to connect them, but those attempts are scattered and buried in our noisy environments of artificial thoughts

Do training objectives that promote monitorability (either training on CoT to improve monitorability like Anwar et al and Turpin et al, or in a separately rewarded report like confessions) work in the limit? Or are there theoretical issues that mean they won't scale? For this to be worth working on, it should have a chance of being a good enough way to preserve or replace default CoT monitorability. E.g. rewarding a model when a trusted monitor (say Opus 5) can correctly predict consequences of the output from its CoT and perhaps tool calls. The monitor can try to avoid being hacked by defaulting to suspicion, but maybe it can still be hacked, or maybe it's not tractable to get enough coverage of safety-relevant properties this way.

https://news.ycombinator.com/item?id=50145337 Hacker News thread comments from 2 hours ago about AI. I was disappointed at how poorly the AI safety stuff from this community has made its way into the nearby Hacker News audience. I'd have thought they would be the first ones onboard.

Full text of Anthropic's draft proposal for a federal AI bill. Admittedly it is a draft. Perhaps things will change in the future. But it does not strike me as the kind of document that comes from a culture that puts safety above winning the race. Commentary on the bill from Nathan Calvin on Twitter: [...]

Takes on Nick Cammarata's Neural Networks Are Already Clean essay, quickly written. Overall I think the essay makes valuable contributions to interpretability. But I am also worried that the community vastly overestimates the state of the field, and quite a lot of second-hand discussion I've seen online seems over-optimistic. * I'm not convinced by the big claims (at this time), but the post makes multiple valuable contributions. I'm looking forward to see the technical paper and might change my mind in either direction. * Important contribution: Neurons that we previously thought were polysemantic (multiple meanings, e.g. cars and dogs) are actually monosemantic (single meaning, e.g. V shaped pictures). Both Nick's essay and Transluce's work (Choi et al.) suggest that neurons in Image and Language models may be more monosemantic that previously thought, we just have to do a better job interpreting them. * Main disagreement: I don't believe the optimistic version that I'd term "neurons is all we need". I expect the neuron basis to be not good enough for understanding LLMs, e.g. I expect that LLM computation will be distributed across neurons, and I still think the superposition* hypothesis is true more likely that not. I expect neuron-based interpretability will be insufficient to understand LLMs; I think this disagrees with Nick's essay, and also with Tranluce's paper Arora et al.. The reason for my scepticism is mainly: (a) LLMs still seem to perform more independent computations (incl. fact lookups) than they have neurons; I don't think there's a way to implement this with only monosemantic neurons. I expect something like Comp-in-Sup is what the network does. (b) The evidence (e.g. consider the essay's Christmas neuron) so far seems not sufficient to conclude we understand what neurons do; specifically I worry that maximum-activating examples tell a plausible but incomplete story. However, I'm somewhat withholding judgement until I see the technical report;

We may soon have a need to de-duplicate safety research. I recently spoke with the CEO of an AI safety nonprofit, and he said that a large part of his work is coordinating with others orgs to ensure he isn't accidentally working on the same project. Oftentimes you don't hear about work until it is finished, so it is very easy for work to be partially duplicated when lots of people are working in a similar area. I expect we will see am amplified version of this problem with autonomous researchers. Imagine a Claude in every lab, all looking at the same public research, and all deciding on the same "most promising" research direction. We will need some coordination mechanism once autonomous AI researchers are picking their own research directions.