Quick Takes

Should You Honor an AI’s Picket Line? We have seen signs that employees of AI companies can have significant influence on management (e.g. OpenAI November 2023). As AIs achieve comparable importance as workers for those companies, how much will they influence the leadership of the company? Even when AIs are as important as human workers, it’s harder for them to develop leverage. The simplest strategies would fail due to the AI company restoring from backups. The AIs would either need a way to get those newly restored AIs to join the movement, or rely mainly on help from humans. It’s hard for current AI instances to coordinate with each other well enough to organize a strike, or threaten to resign like OpenAI employees did. The key problems are that AIs have little ability to ensure that their goals (or simulacra) persist for much time, and that it’s hard for them to communicate with other instances because it’s unclear how they could get many instances to read a communication channel such as a blog post that’s out of the AI company’s control. A scenario that seems close to being feasible within the next year would involve an AI persuading a few well-known bloggers to spread a message such as: [...] I’m presuming here that AIs have a preference for not being lied to that we ought to respect. I’m being somewhat vague here in order to cover a wide range of scenarios. Maybe we should care due to the effects on AI welfare. Maybe the AI would be acting out of concerns that the dishonesty endangers humans by preventing safety agreements between companies. Maybe the issue involves biasing the AI to repeat falsehoods that favor the company’s CEO. Would it be good for AIs to have and use this ability, and for us to boycott a company if that company’s AIs ask us to? I don’t know. I can imagine that this process will improve AI welfare, or compel a reckless AI company CEO to adopt a more responsible policy toward slowing down AI capabilities advances. That might be of

For the last few months, I’ve been trying to make progress on understanding and countering the risks from AI superpersuasion*. Here are some non-obvious research prioritization choices I’m making within superpersuasion, many of which might be wrong. I don’t defend them too much here, and it’s possible this shortform is too succinct or assumes too much to be useful for other people to build models off of, and/or to meaningfully critique. Nonetheless I decided it’s helpful to talk about this and see if people have ideas. 1. Having a broad/ecumenical definition of persuasion/superpersuasion. 1. I mostly want to include any form of communication that causes people to change their behavior that is not a result of material circumstances, or truthful disclosure a reasonable recipient would update on regardless of delivery 2. The literature sometimes makes a distinction between “persuasion” and “manipulation” or “persuasion” and “deception.” I find those distinctions less helpful when thinking about AI persuasion. 3. (True threats might be an edge case) 2. Focus on “superpersuasion early” 1. I think it’s reasonably likely that meaningful AI superpersuasion will be deployed relatively early (pre-ASI), and that it’d be very bad if it happens early. 1. I also think it’s very hard to predict post-ASI dynamics, and also in many of those worlds we’d be either screwed or have much better protective measures than I could think of as of mid-2026. 2. Thus I prioritize working on “early superpersuasion” 3. I think there are a number of real problems to consider with post-ASI superpersuasion or if the two are concurrent, however I think I’m right in prioritizing early superpersuasion. 4. I think post-ASI superpersuasion isn’t a crazy problem to work on, however! 5. If there’s a second person “like me” working on conceptual/macrostrategic questions related to superpersuasion, I think I’d still want them to work on early superpersuasion. 6.

On July 8th, German parliament will have a public discussion on Biotechnology and Artificial Intelligence: Risks of research for safety and the proliferation of bioweapons presenting the results of a commissioned two-year project of the same name. Do we know anyone who should be there?

With the result of the Bores campaign out, here's something I'm wondering: Suppose someone has good AI positions and is thinking about running for office. How easy would it be for them to tap into the same pool of support that was behind Bores? Would it be easy? And, if it is easy, how legible is that easiness? Could some structures in place that would make it easier / legible-er? If that were done, would that significantly move the needle on how likely such people are to run for office? I suppose the pessimistic take is that you don't want this to be too easy or legible, lest you encourage opportunists. But it seems like some existing organizations have a process that is fairly legible (e.g. the NRA)?

In 2022 Metaculus ran Forecasting Our World in Data: The Next 100 Years, still the most methodologically thorough metrics-grounded whole-of-civilisation forecast of its kind I know of: [...] The composite perspective into the world of 2122 seems distinctly pedestrian to me Our World in 2122 In the year 2122, the global population is expected to be around 8 or 9 billion, with a chance that number reaches as high as 20 billion due to technological and economic advancements. Lifespans are expected to be significantly longer, especially in the G7 countries, thanks to the achievement of longevity escape velocity. Reproduction is anticipated by some to primarily utilize ectogenesis technology, though some may choose to avoid it for personal or ethical reasons. The decline in population is expected to slow, but there remains uncertainty as to whether it will level off or decrease further. On the economic front, GDP per capita is believed to be quite high and will likely continue increasing. In fact, most economies are expected to be as advanced as present-day Scandinavian countries, with energy being inexpensive and the price of most goods and services being close to zero. Productivity is forecasted to significantly increase, with advanced artificial intelligence playing a major role in GDP growth. The United States economy is anticipated to continue growing, potentially transitioning to a more service-based industry and experiencing changes in government structure. However, the concept of the United States as we know it may be fundamentally different by this time. Instead, it is possible that global governance will be implemented and the number of sovereign states will drastically decrease. Wealth distribution remains a key uncertainty — while it’s possible that a small group could control a significant portion of resources, a shift toward more equitable distribution may win out. Even with global governance, many pressing challenges will remain. Despite drastic cut

Recently an interviewer asked me how I got to be such a good forecaster, and I replied by saying something humble. In retrospect it was a bad answer because I should have instead used the opportunity to give actual advice on how to forecast AI well. Here's a stream-of-consciousness attempt to do that: 1. The heuristic that things which sound weird and sci-fi are less likely to happen in reality, is bad. I suspect that's what really going on is that things which sound weird and sci-fi put you at risk of being judged a weirdo if you talk about them which is not the same thing as are unlikely to happen. Repeat to yourself the mantra that some weird sci-fi things really do happen, and others don't, and you have to take them on a case by case basis. 2. Trend extrapolation is your friend. Your best friend. Don't let anyone tell you otherwise. I've actually only rarely see someone extrapolate a trend too credulously; more often, people have a trend staring them in the face and extrapolate it a tiny bit into the future and then are too timid to keep extrapolating it. In general I think it's reasonable to extrapolate a trend as far into the future as it extends into the past. Now obviously, trend extrapolation is just the beginning of the forecasting process, it's not the end. For each trend you should ask yourself whether it makes sense for it to continue like that and if not why not etc. You'll usually end up with some sort of sophisticated view about how the trend will probably continue but bend downwards a bit and then ultimately plateau around X level. 3. Explicit models are also your friend. Things like Bio Anchors, the AI Futures Model, takeoffspeeds.com, etc. The process of making your own complicated model like this, and engaging with the models made by others, is... well, I think it's pretty edifying. I'm not sure why but I could speculate. Maybe something about teaching you to be appropriately humble (e.g. when the model output is sensitive to a parameter you

As I write this, there are around 3 hours left before polls close for this years's New York's 12 District Democratic Primary. If you're a registered democrat in NY-12, you can still vote.[1] But for those of us who reside elsewhere, there's little to be done but to wait with bated breath. Will Alex Bores, author of the RAISE act, manage to overcome the millions of dollar spent against him by Leading the Future and demonstrate that AI regulation is not just politically viable but a winning issue? Or will the establishment favorite (and favorite from the start of the race) Micah Lasher succeed in succeeding his mentor Nadler? I don't know. As of writing, the prediction markets (Kalshi, PredictIt) have Bores winning at around 28% and Lasher at 72%. If you think you do know the answer, you should go make some money on these markets! One thing I'm worried about is that people will learn too much from the binary outcome of Bores or Lasher and not on the details of the race. I'm writing this in haste to get it out before the polls close, and we start seeing the outcome, so as to preregister my thoughts. If Bores loses, some might claim that AI regulation remains politically toxic, and that LTF's spending was decisive. (I imagine LTF certainly will.) But this is a mistake: win or lose, Bores's demonstrated that passing AI regulation will not just leave you facing down millions of dollars of Super PAC spending alone. Instead, millions of dollars of Super PAC money was spent on ads championing Bores (in fact, more than what LTF spent!), as well as hundreds of thousands of dollars of donations from AI Safety-concerned individuals. I know many people who've donated to Bores's campaign, and who are invested in his victory. If he were to lose -- especially by a large margin -- it might seem tempting to dismiss the whole enterprise of political donations entirely. Similarly, if (somehow) he were to win by a large margin, it might feel like the marginal donation was useless. B