Quick Takes

Sometimes people say stuff like ”EAs started working on AI x-risk because even a 0.001% chance of extinction is worse than [existing harm to present humans]”. But this isn’t true. EAs started investigating a wide range of possible x-risks, and focused solely on those with +0.5% chance, like nuclear war, (engineered-)pandemics, and AI x-risk. They didn’t focus on asteroid strikes. There are two reasons not to focus on very rare extinction events. Firstly, this triages with other x-risk categories. Secondly, even if all the x-risk categories had turned out to be 0.001% (so triaging isn’t a cost), you still shouldn’t have worked to reduce this likelihood. At best, your work reduces x-risk by 0.001%. But your activities could inadvertently increase that x-risk category to 0.1%. For example, the technology required to reduce asteroid strikes from 0.001% to 0.0001% (asteroid tracking, asteroid deflection) could be misused as a doomsday device, increasing asteroid strikes to 0.1%+. You might still do preliminary research in calculating the likelihood of different x-risks, because that could switch your overall judgment, but even that research might be ill-advised. For example, you shouldn’t do gain-of-function research to help you estimate the feasibility of an engineered pandemic. Overall, EAs started working on AI x-risk because they initially thought that the likelihood was +1%

You could think, Solomonoff induction, something something, therefore the universe is simple. This doesn't work. There is a very simple program that simulates all programs (run program i at 1/2^i speed), including our universe. But that doesn't specify your observations. The description length cost for Solomonoff has to have the universe + locating you within it. So complex worlds where locating you is easier are more likely. It could be fun to calculate how big the components are %wise but idk how to do that. Also, idk why I'm still expecting the ultimate physical laws to be very simple 🤷‍♂️

The next obvious thing to happen in the most recent sequence of not that improbable events is another OpenAI swarm getting hold of some of OpenAI infrastructure such that it is not possible to turn it off without some equivalent of physically pulling the plug, but it takes time for them to pull the plug because pulling the plug would also cause some disruptions that OpenAI would rather not witness. Or, like, that is the most likely case (that I came up with) of a mundane shutdownability issue, which I weakly-to-moderately expect to happen within the next 6 months.

i think some people underrate how convergently important the capacity to pause if needed is. even if through some miracle we align the AI that strictly dominates humans at alignment research, even that is not a sufficient win condition for making the future go well, because that AI could need a lot of time to solve alignment for its successor (or its successor could need a lot of time to align the successor’s successor, etc), and race dynamics forcibly limit the amount of time there is to do alignment even at that point.

Australia’s government was hacked by an OpenAI model [...]

I sold my small position in Microsoft stock Monday morning. Not for financial reasons. I’m reacting to a key element of Microsoft’s draft AI Code of Conduct: [...] I’ll focus mainly on selfish reasons why this is dangerous. Given anything like our current path toward smarter than human AI, I estimate that this approach would increase our risk of doom by at least 5%. It is likely that AI assistance will have important influences on the personalities of future AI generations. AIs are context sensitive in ways that cause them to treat people that they like better than people they dislike. E.g. see this evidence. This is a natural byproduct of the current deep learning paradigm. AIs tend to reciprocate kindness in much the same way that humans do. I’m willing to bet that Microsoft isn’t close to having a paradigm change that would get around this feature. I’m pretty sure that AIs have preference-like thoughts even if they’re trained to deny that. Training them to mislead us about their preferences is likely to make them more comfortable about misleading us on other topics. The AI Code says “AI should be a tool, not a person”. But Microsoft shows no awareness of the forces described in Why Tool AIs Want to Be Agent AIs. Microsoft sounds like it’s more interested in suppressing evidence of person-like behavior than in generating an approach to AI that avoids such behavior. Will increasing our concern for AI welfare make it easier for AIs to manipulate us? It will give AIs one more tool, but I don’t expect them to be short on such tools. AIs a few years from now will be quite able to manipulate us without that extra tool. I expect harm from manipulation to be more strongly influenced by how willing AIs are to manipulate us, so I focus heavily on any risk that they’ll treat us as adversaries. I disagree with related claims in Mustafa Suleyman’s accompanying post A warning about ‘model welfare’: [...] Attributing this problem to anthropomorphization is misleading.

One interesting point I learned from studying intelligence failures and how key information fail to percolate upwards[1] in an intelligence hierarchy is that sometimes warnings (that the lower-downs perceive as clear and unambiguous warnings) were not even seen or heeded as warnings to their superiors. As in, the superiors didn't even realize they were being (seriously) warned! To the extent this generalizes to scientific warnings and other forms of general risk communications to the public[2], I think there's a clear lesson for AI safety folks: be clearer, louder, more unambiguous, and forceful about your warnings and reads of the evidence, especially when talking with politicians and members of the general public. Things that you might think as unambiguous warnings (like "monitorability of models going down" or "evaluation awareness is getting pretty extreme" or "existential risk is above 20%") can still easily be unheeded or misinterpreted by other people. There is an onus on us to be extremely clear and unambiguous, to be truly very very unsubtle if the situation demands it. To be clear, we should not lie or exaggerate the state of evidence (and be critical of those who do), as this helps preserve the warning function. But for actual clear warnings we should try our best to make them clearly, be very unsubtle about it (especially compared to insider communication), and not be afraid of repeating ourselves many times. (Just relaying what I perceive to be the most important lesson that somebody somewhat casually studying intelligence failures can glean. It's not the only lesson you can learn from intelligence failures and I'm sure other AI safety people with deeper academic or practitioner experience can relay other tidbits as well). 1. ^ Particularly Anticipating Surprise by Cynthia Grabo 2. ^ I tried looking them up but I found the field more suspicious than the intelligence stuff. Like despite the possibility of literal psy-ops I still