This is a linkpost for https://paulfchristiano.substack.com/p/personal-statement-on-joining-the
New Comment
I've commented on this announcement on twitter and in the Constellation slack (from which I've resigned my membership, as per below). To quickly summarize: the AI safety community is structurally blocked from reliably improving the world unless it becomes much better at handling adversarial dynamics. In particular, the most important lesson we can learn from the last decade is how and why "AI safety" was so heavily captured by AGI companies, and what it would have taken to prevent that. The most basic building block is the willingness to call a spade a spade, and honestly discuss what's going on with AGI companies; I interpret Paul as committing via his association with OpenAI not to do so. Of course Paul is only one part of Constellation, but his worldview has been (and I expect will continue to be) the single biggest influence on the broader strategy that Constellation is pursuing, which suggests that most reasonable thing to do is to disaffiliate.
The text of my tweet:
I am sad and disappointed to hear that Paul is joining OpenAI's board. Being affiliated with OpenAI has historically led AI safety researchers (including both Paul and myself) to behave in low-integrity ways. I pers...
7
1Some notes on public communication, which I expect the LW audience is especially interested in:
- I believe the only legal restrictions on my speech in connection with OpenAI are non-disclosure agreements with OpenAI PBC and my fiduciary duty to the OpenAI Foundation (separately I also have confidentiality obligations to the US government and confidentiality obligations at ARC). The main thing to have in mind when interpreting my public comments is that you shouldn't expect me to say anything based on confidential information.
- A fiduciary duty implies a duty of confidentiality, including confidentiality about the deliberations and process of the board. I think the implications for speech beyond that are mild, but I think it does mean e.g. that it would be inappropriate for me to try to litigate a boardroom conflict via public statements. I have no legal concerns about publicly stating my own beliefs based on public information even when they are about OpenAI or what OpenAI should do.
- I believe I will inevitably be interpreted by some audiences as speaking for the OpenAI nonprofit or speaking with particular authority about OpenAI, even when speaking in a personal capacity. I think it’s
13
Lol. Ah yes, finally, Paul Christiano is toning down his extremist rhetoric!
If anyone who thinks this comment is somehow inappropriate (or doesn't get why it's amusing) would like to explain why, I would welcome that.
One of the top 3 things that has made Paul's career has been writing a lot in public about this subject in a way that is respectable, never rhetorically alarming, and able to serve as an optimistic counterweight those sounding the alarm. So it is funny in the immediate weeks after the NYT has frontpage pieces about rogue AIs committing crimes and Bernie Sanders announcing a bill to ban superintelligence, to see his one tonal note be that he has softened his writing even further.
4
3I strong downvoted the parent as it seemed unproductive and distracting. I think it's important context that Paul is making the tone softer than what it otherwise would have been given he is a director, and it would be useful for people to extrapolate to what the post otherwise would have been.
able to serve as an optimistic counterweight those sounding the alarm
FWIW, I don't think Paul served a large role as an optimistic counterweight to those sounding the alarm, at least in public.
So it is funny in the immediate weeks after the NYT has frontpage pieces about rogue AIs committing crimes and Bernie Sanders announcing a bill to ban superintelligence, to see his one tonal note be that he has softened his writing even further.
I think there are communication compromises you'd typically take when joining the board of a company. I think there is an open question about the right strategy (both tactically and given typical norms) in circumstances like these.
I read this post as relatively stark and concerned compared to Paul's earlier communication. So while the effect of being a director may have softened his writing relative to what it otherwise would have been, other effects (e.g., the immediacy of risk) may be making it significantly more harsh.
3
1-3
I am annoyed by how you suggest that only your subtribe says what it believes. You know how it would be silly to say "One of the top 3 things that has made Eliezer's career has been writing a lot in public about this subject in a way that is rhetorically alarming"? I think your comment is similarly silly.
Recap for those not tracking the details: my understanding is that the nonprofit (“OpenAI Foundation”) owns 26% of the shares in the public-benefit corporation (“OpenAI Group PBC”). However, the non-profit board has special governance rights that give it complete control over who sits on the PBC board: it appoints all of its directors and can replace them at any time.
Confusingly, the two boards have so far been nearly identical, with 9 people sitting on both, including Bret Taylor as Chair of both boards, and Sam Altman also serving on both (being the only PBC executive to do so). Zico Kolter (Professor at CMU and Head of its Machine Learning Department) was the one exception, serving only on the nonprofit board (and as a non-voting "observer" on the PBC board). Paul Christiano now appears to be the second person appointed only to the nonprofit board.
I guess if there are subcommittees, perhaps more people will be announced soon. We’ll see!
43
My understanding is that the board has the de jure formal power to delay releases but whether or not they have the de facto power to do so is unknown until tested.
9
Is the former actually true? Like, do we have formalized bylaws or anything like that that gives them actual de jure formal power? I have sent ChatGPT off on a research thread to help me figure this out...
ChatGPT says we don't have any solid evidence to support that it's in bylaws, and all we have to go off of is that blogpost. I currently would take a bet that the board does not have actually formalized non-overwriteable formal power to delay releases (but think it's still like 25% likely).
I would be interested in @paulfchristiano clarifying, if he can.
Yes I believe they have the relevant formal power. At least it's in the MOU between the California AG and the OpenAI legal counsel.
The SSC has and will continue to have the authority to require mitigation measures—up to and including halting the release of models or AI systems—even, for the avoidance of doubt, where the applicable risk thresholds would otherwise permit release. The NFP will provide advance notice to the Attorney General of any material changes to the SSC’s authority.
1
14
Zico Kolter said this May in sworn testimony (in the Musk v Altman trial) that the SSC has twice delayed model releases.
(See e.g. here or purchase the full transcript from the court, in which he later says there was "no pushback when the SSC has exercised its authority".)
3
FWIW, my understanding is that the SSC can intervene on internal deployment and certainly the non-profit board has the ability to do this.
Didn't Paul take a government regulator role job while under a secret non-disparagement agreement with OpenAI? If so, pretty dumb move if you ask me.
(Added: More specifically, is a low-integrity thing to do, which worries me greatly.)
1I haven’t addressed it publicly. Here’s a timeline and some root cause analysis:
- When I left OpenAI I negotiated to make sure I could use everything I had learned about safety in my future work. This was a supplement to the normal severance agreement. I had a lawyer review the negotiated supplement but neglected to have them review the main agreement, which I think is the most important root cause. It’s also just a kind of extreme mistake and I’m not sure how I made it.
- I did review the standard severance agreement myself, and looked for anything that would be problematic. It contained a prohibition on making “disparaging, derogatory, defamatory, libelous, or scandalous statements,” which I failed to flag as problematic. I think that was a mistake.
- I vaguely remember or can reconstruct some of my reasoning here though it was a long time ago. For the purpose of understanding how I made the mistake, here are some things that I think were going on:
- I compared the agreement to another severance agreement (from Google) which had similar terms. I mistakenly thought that signing severance agreements was much more common than it actually is. This is related to OpenAI being my first job out of
719
NIST (of which CAISI is a part) is not a regulator.
I am excited to be joining the OpenAI nonprofit board, serving on the Safety and Security Committee to support safety oversight.
Based on the recent trajectory of capabilities and the continued difficulty of alignment, I now believe there is a meaningful risk that rapid acceleration in AI capabilities leads to catastrophic and irreversible loss of control in the very near term.[1] I do not think that the AI industry in general, including OpenAI, is currently on track to reduce this risk to an acceptable level. I’m joining because I believe that if OpenAI rises to the occasion we could significantly reduce risk.
The SSC has an important and challenging role in overseeing risk management at OpenAI, and I hope to help provide expertise and assistance in a critical moment. My joining is not an endorsement or criticism of OpenAI’s safety practices in particular; I hope that all frontier companies strengthen safety oversight and I am excited to work on this at OpenAI. I believe that the rest of the world should judge OpenAI, and all AI developers, by externally verifiable behavior and results.
In the rest of this post, I'll explain why I believe loss-of-control risk is now acute and how I think about the current situation in the AI industry.
First, automated AI R&D could lead to a very rapid acceleration in AI capabilities very soon. OpenAI has predicted that we might have capabilities sufficient to fully automate AI research within 18 months; my personal forecast is extremely uncertain and I think it could easily take anywhere from several months to several years.
Full automation of AI R&D means that improvements in training and algorithms can directly increase the quality and quantity of automated AI researchers available to do additional research. Existing evidence is very uncertain but suggests that this positive feedback loop might be strong enough to overcome diminishing returns and compute bottlenecks, leading to a rapid intelligence explosion. If this happens, then within six months of full AI R&D automation we could see more algorithmic progress than has occurred since the development of the Transformer nearly a decade ago. I believe this would result in superintelligent AI systems.
Second, we currently train our AI agents with RL to get as much reward as they can. It has long seemed theoretically possible that this could motivate AI agents to undermine human control, seek power and resources, and cover up their tracks in pursuit of misaligned goals correlated with reward. Public evidence from recent incidents suggests that this is not just a theoretical possibility.
An intelligence explosion would greatly exacerbate risks from misalignment, both by making the technical problem of alignment even more difficult and by rapidly raising the stakes for failure. Many researchers and leaders at OpenAI and across the industry have expressed concern that rapid recursive self-improvement is not consistent with safe development; I resonated with this recent post by OpenAI's chief scientist Jakub Pachocki on this topic.
If we build superintelligence without more robust alignment I expect we will permanently lose control of it. If that happens then most people could die. I believe we would need domestic and international coordination to ensure global consistency and reduce risk to an acceptable level.
That said, frontier AI developers have a lot of power to unilaterally improve the situation and lay the groundwork for stronger coordination. Developers can improve safety mitigations (including slowing development as necessary), transparently share evidence about risk and the effectiveness of their mitigations, and work towards shared safety standards.
I am encouraged by other members of the SSC, as well as the rest of the board and leadership, taking these issues seriously. I look forward to working with them to help OpenAI raise the bar for its safety practices.
If I had to quantify my uncertainty I would estimate an all-things-considered risk of 4% over the next year and 15% over the next three years. These numbers are a way of stating my subjective beliefs and communicating roughly how big I think the problem is, not a claim to have a model that produces precise or stable estimates.