This makes it even clearer that Altman’s claims of ignorance were lies – he cannot possibly have believed that former employees unanimously signed non-disparagements for free!

This is still quoting Neel, right? Presumably you intended to indent it.

Have you looked through the FLI faculty listed there?
How many seem useful supervisors for this kind of thing? Why?

If we're sticking to the [generate new approaches to core problems] aim, I can see three or four I'd be happy to recommend, conditional on their agreeing upfront to the exploratory goals, and that publication would not be necessary (or a very low concrete number agreed upon).

There are about ten more that seem not-obviously-a-terrible-idea, but probably not great (e.g. those who I expect have a decent understanding of the core problems, but basically aren't working on them).

The majority don't write anything that suggests they know what the core problems are.

For almost all of these supervisors, doing a PhD would seem to provide quite a few constraints, undesirable incentives, and an environment that's poor.
From an individual's point of view this can still make sense, if it's one of the only ways to get stable medium-term funding.
From a funder's point of view, it seems nuts.
(again, less nuts if the goal were [incremental progress on prosaic approaches, and generation of a respectable publication record])

A few points here (all with respect to a target of "find new approaches to core problems in AGI alignment"):

It's not clear to me what the upside of the PhD structure is supposed to be here (beyond respectability). If the aim is to avoid being influenced by most of the incentives and environment, that's more easily achieved by not doing a PhD. (to the extent that development of research 'taste'/skill acts to service a publish-or-perish constraint, that's likely to be harmful)

This is not to say that there's nothing useful about an academic context - only that the sensible approach seems to be [create environments with some of the same upsides, but fewer downsides].

I can see a more persuasive upside where the PhD environment gives:

• Access to deep expertise in some relevant field.
• The freedom to explore openly (without any "publish or perish" constraint).

This seems likely to be both rare, and more likely for professors not doing ML. I note here that ML professors are currently not solving fundamental alignment problems - we're not in a [Newtonian physics looking for Einstein] situation; more [Aristotelian physics looking for Einstein]. I can more easily imagine a mathematics PhD environment being useful than an ML one (though I'd expect this to be rare too).

This is also not to say that a PhD environment might not be useful in various other ways. For example, I think David Krueger's lab has done and is doing a bunch of useful stuff - but it's highly unlikely to uncover new approaches to core problems.

For example, of the 213 concrete problems posed here how many would lead us to think [it's plausible that a good answer to this question leads to meaningful progress on core AGI alignment problems]? 5? 10? (many more can be a bit helpful for short-term safety)

There are a few where sufficiently general answers would be useful, but I don't expect such generality - both since it's hard, and because incentives constantly push towards [publish something on this local pattern], rather than [don't waste time running and writing up experiments on this local pattern, but instead investigate underlying structure].

I note that David's probably at the top of my list for [would be a good supervisor for this kind of thing, conditional on having agreed the exploratory aims at the outset], but the environment still seems likely to be not-close-to-optimal, since you'd be surrounded by people not doing such exploratory work.

RFPs seem a good tool here for sure. Other coordination mechanisms too.
(And perhaps RFPs for RFPs, where sketching out high-level desiderata is easier than specifying parameters for [type of concrete project you'd like to see])

Oh and I think the MATS Winter Retrospective seems great from the [measure a whole load of stuff] perspective. I think it's non-obvious what conclusions to draw, but more data is a good starting point. It's on my to-do-list to read it carefully and share some thoughts.

I agree with Tsvi here (as I'm sure will shock you :)).

I'd make a few points:

1. "our revealed preferences largely disagree with point 1" - this isn't clear at all. We know MATS' [preferences, given the incentives and constraints under which MATS operates]. We don't know what you'd do absent such incentives and constraints.
   1. I note also that "but we aren't Refine" has the form [but we're not doing x], rather than [but we have good reasons not to do x]. (I don't think MATS should be Refine, but "we're not currently 20% Refine-on-ramp" is no argument that it wouldn't be a good idea)
2. MATS is in a stronger position than most to exert influence on the funding landscape. Sure, others should make this case too, but MATS should be actively making a case for what seems most important (to you, that is), not only catering to the current market.
   1. Granted, this is complicated by MATS' own funding constraints - you have more to lose too (and I do think this is a serious factor, undesirable as it might be).
3. If you believe that the current direction of the field isn't great, then "ensure that our program continues to meet the talent needs of safety teams" is simply the wrong goal.
   1. Of course the right goal isn't diametrically opposed to that - but still, not that.
4. There's little reason to expect the current direction of the field to be close to ideal:
   1. At best, the accuracy of the field's collective direction will tend to correspond to its collective understanding - which is low.
   2. There are huge commercial incentives exerting influence.
   3. There's no clarity on what constitutes (progress towards) genuine impact.
   4. There are many incentives to work on what's already not neglected (e.g. things with easily located "tight empirical feedback loops"). The desirable properties of the non-neglected directions are a large part of the reason they're not neglected.
   5. Similar arguments apply to [field-level self-correction mechanisms].
5. Given (4), there's an inherent sampling bias in taking [needs of current field] as [what MATS should provide]. Of course there's still an efficiency upside in catering to [needs of current field] to a large extent - but efficiently heading in a poor direction still sucks.
6. I think it's instructive to consider extreme-field-composition thought experiments: suppose the field were composed of [10,000 researchers doing mech interp] [10 researchers doing agent foundations].
   1. Where would there be most jobs? Most funding? Most concrete ideas for further work? Does it follow that MATS would focus almost entirely on meeting the needs of all the mech interp orgs? (I expect that almost all the researchers in that scenario would claim mech interp is the most promising direction)
   2. If you think that feedback loops along the lines of [[fast legible work on x] --> [x seems productive] --> [more people fund and work on x]] lead to desirable field dynamics in an AIS context, then it may make sense to cater to the current market. (personally, I expect this to give a systematically poor signal, but it's not as though it's easy to find good signals)
   3. If you don't expect such dynamics to end well, it's worth considering to what extent MATS can be a field-level self-correction mechanism, rather than a contributor to predictably undesirable dynamics.
      1. I'm not claiming this is easy!!
      2. I'm claiming that it should be tried.

Detailing what job and funding opportunities should exist in the technical AI safety field is beyond the scope of this report.

Understandable, but do you know anyone who's considering this? As the core of their job, I mean - not on a [something they occasionally think/talk about for a couple of hours] level. It's non-obvious to me that anyone at OpenPhil has time for this.

It seems to me that the collective 'decision' we've made here is something like:

• Any person/team doing this job would need:
  • Extremely good AIS understanding.
  • To be broadly respected.
  • Have a lot of time.
• Nobody like this exists.
• We'll just hope things work out okay using a passive distributed approach.

To my eye this leads to a load of narrow optimization according to often-not-particularly-enlightened metrics - lots of common incentives, common metrics, and correlated failure.

Oh and I still think MATS is great :) - and that most of these issues are only solvable with appropriate downstream funding landscape alterations. That said, I remain hopeful that MATS can nudge things in a helpful direction.

For reference there's this: What I learned running Refine 
When I talked to Adam about this (over 12 months ago), he didn't think there was much to say beyond what's in that post. Perhaps he's updated since.

My sense is that I view it as more of a success than Adam does. In particular, I think it's a bit harsh to solely apply the [genuinely new directions discovered] metric. Even when doing everything right, I expect the hit rate to be very low there, with [variation on current framing/approach] being the most common type of success.

Agreed that Refine's timescale is clearly too short.
However, a much longer program would set a high bar for whoever's running it.
Personally, I'd only be comfortable doing so if the setup were flexible enough that it didn't seem likely to limit the potential of participants (by being less productive-in-the-sense-desired than counterfactual environments).

So no, not disincentivizing making positive EV bets, but updating about the quality of decision-making that has happened in the past.

I think there's a decent case that such updating will indeed disincentivize making positive EV bets (in some cases, at least).

In principle we'd want to update on the quality of all past decision-making. That would include both [made an explicit bet by taking some action] and [made an implicit bet through inaction]. With such an approach, decision-makers could be punished/rewarded with the symmetry required to avoid undesirable incentives (mostly).
Even here it's hard, since there'd always need to be a [gain more influence] mechanism to balance the possibility of losing your influence.

In practice, most of the implicit bets made through inaction go unnoticed - even where they're high-stakes (arguably especially when they're high-stakes: most counterfactual value lies in the actions that won't get done by someone else; you won't be punished for being late to the party when the party never happens).
That leaves the explicit bets. To look like a good decision-maker the incentive is then to make low-variance explicit positive EV bets, and rely on the fact that most of the high-variance, high-EV opportunities you're not taking will go unnoticed.

From my by-no-means-fully-informed perspective, the failure mode at OpenPhil in recent years seems not to be [too many explicit bets that don't turn out well], but rather [too many failures to make unclear bets, so that most EV is left on the table]. I don't see support for hits-based research. I don't see serious attempts to shape the incentive landscape to encourage sufficient exploration. It's not clear that things are structurally set up so anyone at OP has time to do such things well (my impression is that they don't have time, and that thinking about such things is no-one's job (?? am I wrong ??)).

It's not obvious to me whether the OpenAI grant was a bad idea ex-ante. (though probably not something I'd have done)

However, I think that another incentive towards middle-of-the-road, risk-averse grant-making is the last thing OP needs.

That said, I suppose much of the downside might be mitigated by making a distinction between [you wasted a lot of money in ways you can't legibly justify] and [you funded a process with (clear, ex-ante) high negative impact].
If anyone's proposing punishing the latter, I'd want it made very clear that this doesn't imply punishing the former. I expect that the best policies do involve wasting a bunch of money in ways that can't be legibly justified on the individual-funding-decision level.