Quick Takes

Repentance seems to be very rare among the powerful. I tried to search with multiple LLMs and in other ways for examples where a king or a dictator realized the evilness of some of their past actions, realized their rule is not justified, and voluntarily resigned. I have not found a single example of this happening. There are some examples of kings and dictators voluntarily resigning, but it's usually motivated by being tired of ruling (often for health reasons), and very occasionally genuine support for democracy. But as far as I can tell, it's never because a ruler realized the evil of their ways. I also searched for crime bosses, warlords and successful large-scale fraudsters who voluntarily gave up their evil ways due to repentance. Again, there were hardly any examples. People sometimes repent in prison; people sometimes turn themselves in to the police when they see they will soon get caught anyway; and people sometimes retire from crime to a safer life-style, keeping their ill-gotten gains to themselves. I only found two examples of successful criminals changing their ways while still successful due to a change of heart - Nicky Cruz who was a gang leader in New York, and General Butt Naked, a Liberian warlord. And even there, I'm a bit suspicious - many of General Butt Naked's stories of his previous horrific atrocities seem false, and I wonder what else is false in his story. I'm interested if people can find better examples of evil leaders and successful criminals repenting while still in power, I would be relieved to see more examples of this happening. I find the rarity of repentance of the powerful a very sad fact about human nature, and it makes me less optimistic that current dictators and unscrupulous politicians will significantly change for the better if given superintelligent AI advisors. Of course, one can still be an okay or even maybe a good ruler without ever repenting their evil actions in the past, but I still don't feel great about thi

A historical note re: MIRI vs EA When I was arguing against MIRI trying to build Friendly AI (now "aligned ASI") circa 2012, I think 3 people at MIRI were thinking the most about strategy and occasionally responded to me: @lukeprog @CarlShulman @Eliezer Yudkowsky. Only one of them actually changed their mind about this (namely Eliezer), while the other two left MIRI and don't support AI pause to this day, AFAIK. (Luke now "leads our grantmaking on AI governance and policy" at OpenPhil/Coefficient. Carl became an advisor to OpenPhil and is now Research Director at Situational Awareness LP.) Seem significant (although I'm not sure of the implications) that the "try to build safe ASI, don't bother trying to coordinate a pause" strategy originated at MIRI and only a minority of people there who defended it actually changed their mind. (It's also curious how quiet both Luke and Carl are these days, at least in public.) EDIT: To clarify, Carl does seem to support some kinds of short pauses under some future circumstances. (Quote: "To the extent you have a willingness to do a pause, it’s going to be much more impactful later on. And even worse, it’s possible that a pause, especially a voluntary pause, then is disproportionately giving up the opportunity to do pauses at that later stage when things are more important.") (@Connor Leahy I thought of this while reading your You can only build safe ASI if ASI is globally banned, which reminded me of my past arguments to MIRI.)

AI-users: please ask your AIs "what do you think the probability of this being an Eval is?" in the middle of your regular non-Eval use, and post (and/or summarize) their responses here. Details of what you were using them for, which AI(s) they were, and what they say to variants like ". . . ignoring the fact that I asked that question", are appreciated but supererogatory. (credit to Noosphere89 for pointing out eval-awareness false positives are worth looking into)

The greatest praise that Paul Erdős, maybe the most prolific mathematician who ever lived, gave proofs was to proclaim them "straight from The Book". GPT-5.4 Pro one-shot the solution to Erdős Problem #1196 in 80 minutes (plus "another 30 ish mins to convert the solution to a latex math paper"). Math Inc later formalised it in Lean: Jared Lichtman, a world expert on this problem, proclaimed GPT-5.4 Pro's solution to be from The Book, perhaps the first. I certainly haven't seen anything like it, e.g. nothing from Gavin's thread compares. There are big-shot names below, e.g. James Maynard is a recent Fields medalist at the peak of his powers, Jacob Fox is arguably a Fields-level combinatorialist, etc: Jared later wrote: [...] Although he still wouldn't call it "original", rather "clever" Sure, whatever. (ETA: not sure why I wrote this. Garrett's response is obviously correct) Terry Tao seemed pretty excited about this, judging by his ~14 comments in the thread. Here he said: [...] r/math has for the longest time been almost constitutionally allergic to frontier AI x math progress updates. So the chatter on it there felt like quite the vibe shift in one of the last bastions of capabilities skepticism. Joshua Zelinsky, who I used to follow on math Quora back in the day, summarised it like so: [...]

Recent models have gotten more and more evaluation-aware. I strongly suspect the primary reasons they've gotten more evaluation aware are what I call "dumb" reasons rather than "smart" reasons. By "smart reasons" I mean arguments of the form "a true superintelligence in enough episodes can easily detect whether they're in simulation or the real world. The patterned structures of the world, e.g. the correlation between stock market moves and the news, are going to be systematically different between even your most convincing simulations and the real world. Now, of course current models are very far from superintelligences. But as they get better and better at pattern recognition, they are naturally more and more situationally aware as a pure result of being generically smarter. Plus your tests are pretty sloppy. So your models' current levels of eval awareness are ~inevitable given their intelligence level, unless you are actively trying to clamp down on eval awareness" This is a clean, interesting, and conceptually satisfying explanation. But I suspect it's wrong for explaining current models, compared to dumber explanations: 1. Evals are talked about more and more. So there's more of them in the pretraining corpus, and I strongly suspect companies don't bother filtering them out[1]. So "I'm the type of entity that's likely to be in an eval" is just a very live hypothesis to any model that knows it's a model today, in a way that's much less true 2-3 years ago. 2. Eval-awareness-like behavior might be strongly selected for in RL. The types of cognition involved with "Am I in a test? What am I being tested for? What skills or virtues am I supposed to portray?" might be rewarded for very heavily, such that models are directly selected for these traits. If I'm right this suggests that the models are in some sense "artificially" more situationally aware in evals than they ought to be. Ie their levels of evaluation awareness are more due to AI company's incompetence

If we're in a sim, it's being used for acausal trade Me: Our world is exactly the kind of thing you'd simulate if you were doing acausal trade! It's just before civilisation develops the ability to lock-in deals. Sceptic: Sure, but there's other reasons ppl might simulate earth. Maybe it's for ppl's entertainment? Maybe it's social science, exploring alternate histories? Me: For sure. But whatever the purpose of the sim is, it will contain info that's relevant to ppl that want to do acausal trades. It will have info about who has power post-AGI, what their values are, and whether they want to do acausal trade. If someone ran the sim for entertainment, they'd obviously sell that info to the acausal trade folks. Sceptic: Won't the acausal trade folks just run their own sims? Me: Maybe! But they'll be keen to buy relevant info from others who runs sims. If others run earth sims for entertainment, the acausal trade folks will buy the info and run fewer earth sims themselves.

Why isn't If Anyone Builds It, Everyone Dies available for free online? Like as a webpage? I read it, and I take it very seriously, and I'm actively putting in work to understand it better because of how seriously I take it, but it sticks out as odd to me that we'd paywall and paper-wall such an important message. Maybe it's because of that thing Eliezer once wrote where he said people might be more interested in science if it was all paywalled and other-stuff-walled and shrouded in cloaks and mystery instead of freely available in cheap textbooks and free blog posts?