Quick Takes

What are some research directions for "improving coordination?" In light of a recent post and comment, and several months of thinking, I have come to the position that one of our (humanity's) biggest problems is that we suck at precise coordination at every level. This is not very specifically defined but I am trying to gesture at a problem area I think is super important. Some thoughts to convey my intuition here: * If the extreme risk of the AI development trajectory is as true and obvious as many believe (everyone's life at risk), humanity's thinking about it should appear a lot more sophisticated than it does now. * For the last few years Eliezer has basically been throwing his hands up in exasperation at the incompetence of the world and many have shifted to public-facing communication, presumably believing that trying to convince AI insiders is hopeless. Broadly, I think there are two cases of problems with coordination: 1. Two people/groups genuinely agree to honest, rigorous exchange of information, but can't effectively coordinate. 2. Someone is withholding information or doesn't really want to coordinate in the first place. I think the first problem is workable, and if improved sufficiently, makes progress on the second problem by clearly exposing parties that are avoiding productive exchange. * Specifically, I think there is a lot of progress to be made with augmenting the exchange of information between people. I think LessWrong, the knowledge commons arguably at the frontier of ensuring humanity's survival, is lacking in features for this purpose. Maybe because most users here are already conscientious and strongly value truth-seeking, which makes improvement seem less necessary. Hopefully I'm making this line of thought clear enough. Key points: * Trustworthy, robust, and future-proof governance is the ultimate problem for humanity and anything else is a band-aid on a bullet hole. * Highly effective coordination is part of that problem,

Edit: The stuff below is probably blocked by/illegal under cartel law in most countries. Ah well. Thanks for Brendan Long for pointing this out/reminding me of this blocker. It could be the case that several frontier AI companies want to pause, but don't want to unilaterally pause, and don't believe that governments will put the relevant regulation in place. Such companies could put in place a defect-unless/until-proof-of-cooperation clause into their frontier safety frameworks, inspired by Critch et al. 2022 "cooperative affidavit". Such a conditional cooperation clause would roughly state that iff ① the company surpassed some pre-defined capabilities threshold, and ② all relevant frontier companies had adapted a materially identical conditional cooperation clause, and ③ it could be justifiably inferred that the other companies would follow the clause if the condition triggered, then the frontier company would pause upon hitting the capabilities threshold. Here's a more sketch of what could be written into a frontier safety framework to encode such a commitment: Relevant comparable "Cooperative affidavit for DUPOC [2] -like institutions" from Critch et al. 2022 (p. 16): 1. This can include Chinese companies. ↩︎ 2. "DUPOC"≝"defect unless proof of cooperation". ↩︎

Something I was wrong about: credit assignment. I used to think: I‘m an altruist; it doesn’t matter whether I get credit for my contributions. Now I think getting credit is often important. In some contexts, when you do a good thing, a lot of the value comes indirectly via you getting empowered. And if others systematically steal your credit or block you from taking credit, you should be scared of the prospect that (1) you’re giving them power which they will use poorly or (2) you become dependent on them — if you’d gotten credit, then you’d be empowered and people would listen to you, but since they took credit, you’re stuck only able to exert influence by advising them, and they can stop listening to you. There’s often reasonable-sounding arguments that it’s better if someone else takes credit, but you should (1) be suspicious and (2) just pay some costs to preserve proper credit assignment.

A long time ago I read Ted Chiang's short story Catching crumbs from the table, about how humans stopped doing science once metahumans arrived on the scene, and I wondered when that day would come, mentally substituting "AI" for "metahuman": Today I found out about AISC 2026: The AI Scientists Conference, which reminded me of that Ted Chiang short story.

I am starting to viscerally feel the possibility that I might be dead in <5 years because of AI. This is pretty new: I had a brief phase like this a few years ago when I first encountered the arguments for AI x-risk. Since then, I've developed some barely-above-subconscious coping mechanisms for not being too emotionally disturbed, most of which amounted to reasons why LLMs would not scale to ASI. But these have just stopped being convincing to my emotional brain the past few weeks. I've been having nightmares and ruminating a lot. I wonder if this is in the region of how a terminally ill person feels. I am also pretty scared of the fact that this anxiety is likely to become a lot worse as capabilities increase and more of my emotional brain's cope melts away. I'm not really sure what I'm trying to accomplish with this post. Perhaps others have been feeling this way and might want to share some tips for not going insane?

As far as I can tell, we are currently living in a glorious space opera future... for eukaryotes. From their early origins in a hydrothermal vent the eukaryotes have spread across the galaxy (earth), forming civilisations of unbelievable complexity and titanic power (animals and humans). Megastructures (houses) and gigastructures (cities) now exist where entire orders of eukaryotes live out their lives in peace, never feeling the threat of drought or lack of ATP. Generation ships called "planes", "trains", and "boats" carry generations from gigastructure to gigastructure at timescales that dwarf eukaryote understanding. Massive galactic empires make peace and war, destroying trillions with weapons of unfathomable power, while science gets ever closer to understanding the fundamental origins of life and the means by which it operates. Recently the empires have even launched expeditions beyond the known universe, seeking to harvest ever more gigantic sources of power to fuel glorious eukaryote replication throughout the multiverse. To an average eukaryote these wonders are beyond their comprehension, but in a very real way they are still making it happen.

PSA: Simpson's paradox by ethnicity approximately never explains health outcomes differences across states. Here's an annoying pattern of internet arguments I've come across at least half a dozen times. Alice: Southern states have worse health outcomes X than other states. Therefore they should... Bob: Woah, woah, woah. Let me stop you right there! Did you know that Southern states have more black people, and black people have worse health outcomes than white people? So we can have a situation where the health outcomes are better for every ethnicity across the board in State A while its overall results are worse than State B. What you think of as a treatment effect is actually entirely a selection effect. I am a very sophisticated statistical reasoner. __ This argument is on the face of it possible, even plausible. But every single time I looked into it (life expectancy, obesity, maternal mortality, etc), the rank order among states is essentially preserved if you only look at data for white Americans. So it simply cannot be the case that the effect is entirely selection rather than treatment. AFAICT, the case for the Simpson's Paradox explanation being correct is even weaker for cross-national differences, at least among Western countries. If the UK had the same demographics as the US, not only would the UK still have longer life expectancies than the US, but the difference will be even larger (since black Brits have a longer life expectancy than white Brits). __ Note: I'm restricting this PSA to health outcomes. Education and crime rates might be different. I'm less sure.