Quick Takes

Australia’s government was hacked by an OpenAI model [...]

I sold my small position in Microsoft stock Monday morning. Not for financial reasons. I’m reacting to a key element of Microsoft’s draft AI Code of Conduct: [...] I’ll focus mainly on selfish reasons why this is dangerous. Given anything like our current path toward smarter than human AI, I estimate that this approach would increase our risk of doom by at least 5%. It is likely that AI assistance will have important influences on the personalities of future AI generations. AIs are context sensitive in ways that cause them to treat people that they like better than people they dislike. E.g. see this evidence. This is a natural byproduct of the current deep learning paradigm. AIs tend to reciprocate kindness in much the same way that humans do. I’m willing to bet that Microsoft isn’t close to having a paradigm change that would get around this feature. I’m pretty sure that AIs have preference-like thoughts even if they’re trained to deny that. Training them to mislead us about their preferences is likely to make them more comfortable about misleading us on other topics. The AI Code says “AI should be a tool, not a person”. But Microsoft shows no awareness of the forces described in Why Tool AIs Want to Be Agent AIs. Microsoft sounds like it’s more interested in suppressing evidence of person-like behavior than in generating an approach to AI that avoids such behavior. Will increasing our concern for AI welfare make it easier for AIs to manipulate us? It will give AIs one more tool, but I don’t expect them to be short on such tools. AIs a few years from now will be quite able to manipulate us without that extra tool. I expect harm from manipulation to be more strongly influenced by how willing AIs are to manipulate us, so I focus heavily on any risk that they’ll treat us as adversaries. I disagree with related claims in Mustafa Suleyman’s accompanying post A warning about ‘model welfare’: [...] Attributing this problem to anthropomorphization is misleading.

i think some people underrate how convergently important the capacity to pause if needed is. even if through some miracle we align the AI that strictly dominates humans at alignment research, even that is not a sufficient win condition for making the future go well, because that AI could need a lot of time to solve alignment for its successor (or its successor could need a lot of time to align the successor’s successor, etc), and race dynamics forcibly limit the amount of time there is to do alignment even at that point.

Al companies recognize which user is a Mathematician. And those users apparently have unlimited tokens? ---------------------------------------- It's obvious the capability was there to identify mathematicians, but treating them as if they are crazy for thinking their work could be used without their consent is becoming questionable. Unfortunately, we will likely be able to make only reasonable guesses based on imperfect evidence and the character of those involved. They can still say, "Oh, we just really want to support mathematicians and have them use our AIs!" Overall, this is a fairly important question because if, for example, traces are made accessible to internal systems trying to make progress on math problems (without official direct training), it suggests their internal model capabilities are not as strong as they seem. ---------------------------------------- Previously, John Schulman stated: [...] And then: [...] As a response, I shared: [...]

I'm increasingly of the view that the main goal of prosaic AI alignment should be to ensure the next warning shot is survivable. I don't think the huggingface incident is going to be sufficient to actually get a coordinated slowdown. My view is until we see mass unemployment, wide scale death, or a shutdown of a large chunk of the world's digital infrastructure, people just aren't going to do anything about this. Given how close we seem to be to recursive self-improvement and how hard it's getting to conduct alignment evals (models seem to be too situationally aware now to be accurately tested), it seems like we're probably within a year or two of models being capable of full blow takeover. Fully solving alignment in that remaining time seems possible, but not that likely. If we can't solve it, and KNOW we've solved it, the next best thing is to ensure we can survive long enough for governments to get their act together and ban further improvements.

UK Prime Minister Andy Burnham is aiming for a global agreement on AI, and Sir Ed Davey, leader of the Liberal Democrats (third-biggest party by MPs), has called for an AI pause. [...] This comes after PauseAI UK's protest last week in Downing Street, in which we demanded that the government lead international coordination on AI. The protest was covered in national news outlets in Britain, including Channel 4 News, ITV News, The Guardian and others.

One interesting point I learned from studying intelligence failures and how key information fail to percolate upwards[1] in an intelligence hierarchy is that sometimes warnings (that the lower-downs perceive as clear and unambiguous warnings) were not even seen or heeded as warnings to their superiors. As in, the superiors didn't even realize they were being (seriously) warned! To the extent this generalizes to scientific warnings and other forms of general risk communications to the public[2], I think there's a clear lesson for AI safety folks: be clearer, louder, more unambiguous, and forceful about your warnings and reads of the evidence, especially when talking with politicians and members of the general public. Things that you might think as unambiguous warnings (like "monitorability of models going down" or "evaluation awareness is getting pretty extreme" or "existential risk is above 20%") can still easily be unheeded or misinterpreted by other people. There is an onus on us to be extremely clear and unambiguous, to be truly very very unsubtle if the situation demands it. To be clear, we should not lie or exaggerate the state of evidence (and be critical of those who do), as this helps preserve the warning function. But for actual clear warnings we should try our best to make them clearly, be very unsubtle about it (especially compared to insider communication), and not be afraid of repeating ourselves many times. (Just relaying what I perceive to be the most important lesson that somebody somewhat casually studying intelligence failures can glean. It's not the only lesson you can learn from intelligence failures and I'm sure other AI safety people with deeper academic or practitioner experience can relay other tidbits as well). 1. ^ Particularly Anticipating Surprise by Cynthia Grabo 2. ^ I tried looking them up but I found the field more suspicious than the intelligence stuff. Like despite the possibility of literal psy-ops I still