One thing I notice when reading 20th century history is that people in the 1900s-1970s had much higher priors than modern people do that the future might be radically different, in either great or terrible ways. For example: * They talked about how WW1 was the war to end all wars. They seriously talked about the prospect of banning war after WW1. Such things now sound hopelessly naive. * Serious people talked very seriously about the possibility of transformative technological change and social change following from it (e.g. Keynes/Russell speculating that people would work way fewer hours in the future). * As a minor example, between 1905-1915 Churchill spent a bunch of time trying to persuade the British government that on current trends, oil-powered ships would soon be way better than coal-powered ships, and the navy should be converted to oil power. I know of ~no recent examples where a major politician's main schtick was being thoughtful about the future of technology and making policy based on it. More generally, it was obvious after WW1 that states needed to be doing futurism and technological development in order to understand the military implications of modern technology. I really feel like the ambient cultural sense among educated Americans is: the future will be kind of like the present, treating it as if something radical will happen is naive. (They sort of say that they think climate change will be apocalyptic, but it feels to me like what they're really imagining is that the world is "enshittified" further, in the same way that it sucks that DoorDash is now expensive, and maybe poor people elsewhere die.) I think this is probably mostly because there's an important sense in which world has been changing more slowly (at least from the perspective of Americans), and the ways in which it's changing feel somehow less real. Someone who was 50 in 1945 had seen the collapse of empires that had lasted centuries, unprecedented wars, the sudden shocking r

Lightcone is doing another fundraiser this year[1]! I am still working on our big fundraising post, but figured I would throw up something quick in case people are thinking about their charitable giving today. Short summary of our funding situation: We are fundraising for $2M this year. Most of that goes into LessWrong and adjacent projects. Lighthaven got pretty close to breaking even this year (though isn't fully there). We also worked on AI 2027 which of course sure had a lot of effects. We do kind of have to raise around this much if we don't want to shut down since most of our expenses are fixed costs (my guess is the absolute minimum we could handle is something like $1.4M). Donors above $2,000 can continue to get things at Lighthaven dedicated to them. Donate here: https://www.every.org/lightcone-infrastructure  (I also just added a fundraising banner. I expect that to be temporary as I don't generally like having ad-like content on post pages, but we happen to be getting a very enormous amount of incoming traffic to Claude 4.5 Opus' Soul Document, and so a banner seemed particularly valuable for a day or two.) 1. ^ Last year's fundraiser: https://www.lesswrong.com/posts/5n2ZQcbc7r4R8mvqc/the-lightcone-is-nothing-without-its-people 

Some ramblings on LW AI rules vs "Avoid output-without-prompt". * I really like the idea of otherwise high-quality posts that are just prompt-and-collapsed-output, actually. I suspect they'll be fairly well upvoted. If not, then downvoters/non-upvoters, please explain why a post that could pass as human-written but which is honest about being ai-written would not get your upvote if it was honest about its origin. * If your post isn't worth your time to write, then it may or may not be worth my time to read; I want to read your prompt to find out. If your prompt is good - eg, asks for density, no floating claims, etc - it likely is worth my time to read. (I wrote the linked prompt entirely by my own word choices.) * I expect that prompt heavily influences whether I approve or disapprove of an AI-written post. Most prompts I expect to see will reveal flaws in the output that would otherwise be hard to spot. Some prompts will be awesome. * My ideal case is human is maker, AI is breaker. I don't usually like AI-as-maker posts where a human has a vague idea and the AI fills it in, because the things AI is still way below human capability at are things I think we need a lot of to do good work. I want the AI's capability to be used to direct human attention to flaws, but not to be the only thing directing human attention to flaws, in case the AI is systematically inclined to miss things for any reason. This is not to say AIs are weak; at this point they're at or near "superhuman, but reachable with an imaginable amount of effort" for most tasks. * If you expect rerolls of the same prompt produce much lower quality output - eg you needed curation, or have additional prompts you don't share, or etc - then sure, don't share prompt. * If you won't share full prompt, perhaps just say "this is AI output, heavily edited". Put it in collapseable sections. What, you aren't brave enough to put your whole post in a collapseable section? * Perhaps convert output into a new pro

The capability evaluations in the Opus 4.5 system card seem worrying. The provided evidence in the system card seem pretty weak (in terms of how much it supports Anthropic's claims). I plan to write more about this in the future; here are some of my more quickly written up thoughts. [This comment is based on this X/twitter thread I wrote] I ultimately basically agree with their judgments about the capability thresholds they discuss. (I think the AI is very likely below the relevant AI R&D threshold, the CBRN-4 threshold, and the cyber thresholds.) But, if I just had access to the system card, I would be much more unsure. My view depends a lot on assuming some level of continuity from prior models (and assuming 4.5 Opus wasn't a big scale up relative to prior models), on other evidence (e.g. METR time horizon results), and on some pretty illegible things (e.g. making assumptions about evaluations Anthropic ran or about the survey they did). Some specifics: * Autonomy: For autonomy, evals are mostly saturated so they depend on an (underspecified) employee survey. They do specify a threshold, but the threshold seems totally consistent with a large chance of being above the relevant RSP threshold. (In particular the threshold is "A majority of employees surveyed think the AI can't automate a junior researcher job AND a majority think uplift is <3x". If 1/4 of employees thought it could automate a junior researcher job that would be a lot of evidence for a substantial chance it could!) * Cyber: Evals are mostly saturated. They don't specify any threshold or argue for their ultimate judgement that the AI doesn't pose catastrophic cyber risk. * Bio: To rule out the CBRN-4 threshold (uplift for moderately resourced state programs, e.g. North Korea), they seem to depend mostly on a text-based uplift trial. The model is extremely close to the relevant threshold and it's unclear how much confidence we should have in this uplift trial. Generally, it seems like the curre

If you live in a universe with self-consistent time loops, amor fati is bad and exactly the wrong approach. All the fiction around this, of course, is about the foolishness of trying to avoid one's fate; if you get a true prophecy that you will kill your father and marry your mother, then all your attempts to avoid it will be what brings it about, and indeed in such a universe that is exactly what would happen. However, a disposition to accept whatever fate decrees for you makes many more self-consistent time loops possible. If on the contrary your stance is "if I get a prophecy that something horrible happens I will do everything in my power to avert it," then fewer bad loops would hypothetically complete, and you're less likely to get the bad prophecy (even though, if you do, you'd be just as screwed, and presumably less miserable about it and foolish-looking than if you had just accepted it from the beginning.) (If you live in a nice normal universe with forward causality this advice may not be very useful, except in the sense that you should also not submit to prophecies, albeit for different reasons.)

My colleagues and I are finding it difficult to replicate results from several well-received AI safety papers. Last week, I was working with a paper that has over 100 karma on LessWrong and discovered it is mostly false but gives nice-looking statistics only because of a very specific evaluation setup. Some other papers have even worse issues. I know that this is a well-known problem that exists in other fields as well, but I can’t help but be extremely annoyed. The most frustrating part is that this problem should be solvable. If a junior-level person can spend 10-25 hours working with a paper and confirm how solid the results are, why don’t we fund people to actually just do that?  For ~200k a year, a small team of early career people could replicate/confirm the results of the healthy majority of important safety papers. I’m tempted to start an org/team to do this. Is there something I’m missing? EDIT: I originally said "over 100 upvotes" but changed it to "over 100 karma." Thank you to @habryka for flagging that this was confusing.

Someone on the EA forum asked why I've updated away from public outreach as a valuable strategy. My response: I used to not actually believe in heavy-tailed impact. On some gut level I thought that early rationalists (and to a lesser extent EAs) had "gotten lucky" in being way more right than academic consensus about AI progress. I also implicitly believed that e.g. Thiel and Musk and so on kept getting lucky, because I didn't want to picture a world in which they were actually just skillful enough to keep succeeding (due to various psychological blockers). Now, thanks to dealing with a bunch of those blockers, I have internalized to a much greater extent that you can actually be good not just lucky. This means that I'm no longer interested in strategies that involve recruiting a whole bunch of people and hoping something good comes out of it. Instead I am trying to target outreach precisely to the very best people, without compromising much. Relatedly, I've updated that the very best thinkers in this space are still disproportionately the people who were around very early. The people you need to soften/moderate your message to reach (or who need social proof in order to get involved) are seldom going to be the ones who can think clearly about this stuff. And we are very bottlenecked on high-quality thinking. (My past self needed a lot of social proof to get involved in AI safety in the first place, but I also "got lucky" in the sense of being exposed to enough world-class people that I was able to update my mental models a lot—e.g. watching the OpenAI board coup close up, various conversations with OpenAI cofounders, etc. This doesn't seem very replicable—though I'm trying to convey a bunch of the models I've gained on my blog, e.g. in this post.)