Quick Takes

Rob Wiblin asked: [...] My response: [...] As additional context, I'll note that I was just talking about misalignment risk and security; I didn't mention considerations relevant to e.g. preventing coups.

I think that the views of superforecasters on AI / AI risk should be basically no update. It seems to me like the main reasons to defer to someone are: 1. They have a visibly good track record on the relevant domain. It has to be the literal domain, because people often have good views on their area of expertise, but crazy views elsewhere. 2. They are highly selected for having good beliefs in the domain. For example, if a mathematician tells me something that seems surprising about their area of expertise, I will tend to strongly believe them, despite not being able to evaluate their reasoning. The general reason for this is because mathematics is a verifiable domain, mathematicians are strongly selected for being correct about math. Other domains I'd basically defer to people in are historians about literal historical facts, physicists about well-established physics results, engineers about how cars work, etc. This consideration weakens as disciplines become less verifiable: I'm not very inclined to defer to philosophers, sociologists, psychologists, etc. 3. They make correct arguments about the domain (and very few incorrect arguments). If it's the case that you can talk to someone and they can consistently make clear rock-solid arguments that change your mind regularly, it is justified to defer to them on bottom line conclusions, even if you can't follow the arguments all the way through. 4. They are much smarter than you and are probably being honest. If someone (or, eventually, an AI) is clearly much smarter than you, and they are being honest (e.g. because they seem like an honest person), then you should probably defer to them substantially. (Of course, this isn't even fully general, e.g. a few hundred years ago, many of the smartest people around were superstitious, which would have led you astray.) Now I'll go through and argue why these don't apply. 1. I think the track record of superforecasters on AI looks quite bad. Superforecasters consistent

I don't feel confused by LLMs seeming very smart while being unable to automate hard work. Sometimes people find it mysterious or surprising that current AIs can't fully automate difficult tasks given how smart they seem. I don't find this very confusing. Current LLMs are just not that "smart" (yet). They compensate using very broad knowledge and strong heuristics that are mostly domain-specific. In other words, they have high crystallized intelligence but lower fluid intelligence. In humans, crystallized and fluid intelligence are very correlated due to limited time for learning and limited capacity for knowledge/memorization, but AIs can train for longer and (for unclear reasons) seem to have much higher capacity for knowledge/memorization. So, AI capabilities are often overestimated by naive comparisons. [1] I do find it somewhat mysterious/surprising that humans have such poor memory despite seemingly having so many parameters, and that some people have much better memory than others. My view is that this crystallized vs fluid distinction is maybe the first principal component of human vs AI differences and this explains a lot of the differences, but not all of them. We can add an additional "component" to our analysis of fluid intelligence by distinguishing between short-run fluid intelligence (we could call this "adaptation") and longer term fluid intelligence (this might effectively be sample efficiency of longer run learning). I think AIs are relatively stronger at short-term fluid intelligence than long-term fluid intelligence. To be clear, this doesn't mean I think extremely powerful AI isn't near: I expect full AI R&D automation [2] in <5.5 years [3] (and it seems 25% likely in <2.5 years). These AIs will likely retain an inhuman profile but will be trained heavily on relevant tasks (while still having much stronger fluid intelligence than current AIs). But pretty soon (0.25-4 years) after full AI R&D automation, I do expect AIs to exceed humans at e

I conducted an exercise at METR to simulate what our work would be like in 2027, when we have 200 hour time horizon AIs. Some observations: * The pace of research was much faster than today, something like 3x. I would guess that speedup goes as time horizon to the 0.3 or 0.4 power, though we didn't run the game with enough fidelity to tell * No time to develop ideas before implementing: Agents implement ideas as soon as you think of them, so rather than ideating for days at a time, you can make an MVP in a couple of hours and revise. If the task isn’t near the limit of agent capabilities, you spend all your time understanding results; if it is, you spend all your time checking its work. * Keeping agents fed overnight: Overnight, agents can do maybe 200 human hours of work, but only for very agent-shaped tasks, so researchers need to deliberately sequence projects such that very long tasks suitable for agents happen overnight, e.g. optimizing a well-defined metric. * Prioritization and organization are bottlenecks: If agents can execute all your ideas nearly as fast as you can prompt them, there’s no point in implementing only your best idea. It might be better to implement your top three ideas all in parallel, but this makes it harder to stay organized. Even with AI-written dashboards to optimally help humans understand, the complexity of projects will probably go up in a way that makes projects much harder to manage. * Anything that takes serial time will no longer happen in parallel with execution, but rather becomes a serial bottleneck. Perhaps the vast majority of total project time will be taken by things like human data, ML experiments, and feedback (from peers, managers, especially external advisors). You can read more at the METR blog.

There's a strong pattern in ratfic of the protagonist "winning" by gaining the power to design a new world order from scratch—i.e. taking over the world. It's a very High Modernist mindset (as I pointed out in a recent tweet). And once you see how crucial this is to the rationalist perspective on what a good future looks like, it's hard to unsee. You might respond: the worlds these protagonists find themselves in are usually so bad that seizing absolute power is in fact the most ethical thing to do. But the worlds didn't have to be that bad! The writers chose to design them that way—I expect in significant part because that provides a narratively compelling backdrop for the thing they wanted to write about, which was their heroes taking over the world. From my current perspective, this feels ominous in a kinda similar way as if I read a bunch of Marxist fiction from the 1800s and noticed that it always ended in a communist revolution. Spoilers for a range of ratfic (unfortunately telling you which ratfic itself constitutes a spoiler): EDIT: the spoiler tags seem not to be working on the images. Mods, can you help? For now I'll add a bunch of space below. Expect many spoilers if you scroll down. And some examples which (at least partially) subvert this trope:

https://en.wikipedia.org/wiki/Chiune_Sugihara

LLMs interact to the worlds only through the context window, so their observations, actions and thoughts all happen on the same interface[1]. (Suggestion on bluesy was that this means LLMs are, in some sense, embodied, which is a neat framing.) I think this could explain some kinds of failure modes (and predict ones that I personally haven't seen yet); where the LLM confuses tokens of one type for the other type →Obs →Thought →Act Obs→ Correct System prompts/context stuffing Prefill attack Thought→ Confabulation Correct Ghost tool call Act→ Misattributed action/"alien hand syndrome" Fictionalized action/autosuggestion Correct * Obs→Thought: LLM observes seeing something in its context, so LLM thinks it thought that (this is both system prompt & sometimes prefill attacks) * Obs→Act: Classic prefill attack territory. * Thought→Obs: Easy one. LLM thinks something, and then LLM believes LLM observed it. (There's a trickier case where LLM thinks something and then LLM believes it remembered that) * Thought→Act: LLM thinks it makes a tool call, then treats that tool call as having been made (this could also be called a confabulated tool call). Kind of rare, I think. * Act→Obs: LLM takes an action, then believes the action was taken by someone else. In the context of a prefill attack the right move. * Act→Thought: LLM acts, then dismisses that action as a thought. Very rare, haven't heard people talking about this one yet. There's some trickiness here with identifying misattributions "prospectively"/"retrospectively" which has something to do with shifting blame, e.g. with Act→Obs it could either be "I'll that action but I can't admit/believe that, let me say I merely saw it" or "Did I take that action? Nay, it's merely an observation!" 1. ^ Memory too, somewhat, though that one's more another axis rather than another type (so Mem(Obs), Mem(Act), Mem(Thought), vaguely, though not definitely).