As I said, I haven't oriented on this subject yet, and I'm talking from my intuition, and I might be about to say stupid things. (And I might think different things on further thought. I think, I 60% to 75% "buy" the arguments that I make here.]

I expect we have very different worldviews about this area, so I'm first going to lay out a general argument, which is intended to give context, and then respond to your specific points. Please let me know if anything I say seems crazy or obviously wrong.

General Argument

My intuition says that in general, governments can only be helpful after the core, hard problems of alignment have been solved. After that point, there isn't much for them to do, and before that point, I think they're much more likely to cause harm, for the sorts of reasons I outline in this comment.

(There is an argument that EAs should go into policy because the default trajectory involves governments interfering in the development of powerful AI, and having EAs in the mix is apt to make that interference smaller and saner. I'm sympathetic to that, if that's the plan.)

To say it more specifically: governments are much stupider than people, and can only do sane, useful things if there is a very clear, legible, common knowledge standard for which things are good and which things are bad.

• Governments are not competent to do things like assess which technical research is promising. Especially not in fields that are as new and confusing as AI safety, where the experts themselves disagree about which approaches are promising. But my impression is that governments are mostly not even competent to do much more basic assessment of things like "which kinds of batteries for electric cars, seem promising to invest in? (Or even physically plausible?)"
• There do appear to be some exceptions to this. DARPA and IARPA seem to well designed for solving some kinds of important engineering problems, via a mechanism that spawns many projects and culls most of them. I bet DARPA could make progress on AI alignment if there were clear, legible targets to try and hit.
• Similarly governments can constrain the behavior of other actors via law, but this only seems useful if it is very clear what standards they should be enforcing. If legislatures freak out about the danger of AI, and then come up with the best compromise solution they can, for making sure "no one does anything dangerous" (from a partial, at best, understanding of the technical details), I expect this to be harmful on net, because it inserts semi-random obstacles in the way of technical experts on the ground trying to solve the problem.

. . .

There are only two situations in which I can foresee policy having a major impact, a non-extreme story, and an extreme story.

The first, non-extreme story is when all of the following conditions hold...

1) Earth experiences a non-local takeoff.

2) We have known, common knowledge, technical solutions to intent alignment.

3) Those technical solutions are not competitive with alternative methods that "cut corners", with regard to alignment, but which do succeed in hitting the operator's goals in the short term.

In this case we know what needs to be done to ensure safe AI, but we have a commons problem: Everyone is tempted to forgo the alignment "best practices" because they're very expensive (in money, or time, or whatever) and you can get your job done without any fancy alignment tech.

But every unit of unaligned optimization represents a kind of "pollution", which adds up to a whimper, or eventually catalyzes a bang.

In this case, what governments should do is simple: tax, or outlaw, unalignment pollution. We still have a bit of an issue in that this tax or ban needs to be global, and free riders who do pollute will get huge gains from their cheaper unaligned AI, but this is basically analogous to the problem of governments dealing with global climate change.

But if any of the above conditions don't hold, then it seems like our story starts to fall apart.

1) If takeoff is local, then I'm confused about how things are supposed to play out. Deep Mind (or some other team) builds a powerful AI system that automates AI research, but is constrained by the government telling them what to do? How does the government know how to manage the intelligence explosion better than the by-definition, literal leaders of the field?

I mean, I hope they use the best alignment technology available, but if the only reason why they are doing that is "its the law", something went horribly wrong already. I don't expect constraints made by governments to compensate for a team that doesn't know or care about alignment. And given how effective most bureaucracies are, I would prefer that a team that does know and care about alignment not be needing to work around the constraints imposed by a legislature somewhere.

(More realistically, in a local takeoff scenario, it seems plausible that the leading team is nationalized, or there is otherwise a very close cooperation between the technical leaders of that team, and military (and political?) leaders of the state, in the style of the Manhattan project.

But this doesn't look much like "policy" as we typically think about it, and the only way to influence this development would be to be part of the technical team, or be one of the highest ranking members of the military, up to the president him/herself. [I have more to say about the Manhattan project, and the relevance to large AI projects, but I'll go into that another time.])

But maybe the government is there as a backstop to shutdown any careless or reckless projects, while the leaders are slowly and carefully checking and double checking the alignment of their system? In which case, see the extreme scenario below.

2) If we don't have solutions to intent alignment, or we don't have common knowledge that they work, then we don't have anything that we can carve off as "fine and legal" in contrast to the systems that are bad and should be taxed or outlawed.

If we don't have such a clear distinction, then there's not much that we can do, except ban AI, or ML entirely (or maybe ban AI above a certain compute threshold, or optimization threshold), which seems like a non-starter.

3) If there aren't more competitive alternatives to intent-aligned systems, then we don't need to bother with policy: that natural thing to do is to use intent-aligned systems.

The second, extreme scenario in which government can help:

We're establishing a global coalition that is going to collectively build safe AI, and we're going to make building advance AI outside of that coalition illegal.

Putting the world on lock-down, and survailing all the compute to make sure that no one is building an AI, while the global coalition figures out how to launch a controlled, aligned intelligence explosion.

This seems maybe good, if totally implausible from looking at today's world.

Aside from those two situations, I don't see how governments can help, because governments are not savvy enough to do the right thing in technical complicated topics.

Responding to your specific points

• Funding safety research

This is only any use at all if governments can easily identify tractable research programs that actually contribute to AI safety, instead of have "AI safety" as a cool tagline. I guess that you imagine that that will be the case in the future? Or maybe you think that it doesn't matter if they fund a bunch of terrible, pointless research if some "real" research also gets funded?

• Building aligned AI themselves?

What? It seems like this is only possible if the technical problem is solved and known to be solved. At that point, the problem is solved

• Creating laws that prevent races to the bottom between companies (e.g. "no AI with >X compute may be deployed without first conducting a comprehensive review of the chance of the AI adversarially optimizing against humanity")

Again, if there are existing, legible standards of what's safe and what isn't this seems good. But without such standards I don't know how this helps?

It seems like most of what makes this work is inside of the "comprehensive review"? If our civilization knows how to do that well, then having the government insist on it seems good, but if we don't know how to do that well, then this looks like security theater.

• Monitoring AI systems (e.g. "we will create a board of AI investigators; everyone making powerful AI systems must be evaluated once a year")

This has the same issue as above.

[Overall, I something like 60% to 75% believe the arguments that I outline in this comment.]

(Some) cruxes:

• [Partial] We are going to have clear, legible, standards for aligning AI systems.
• We're going to be in scenario 1 or scenario 2 that I outlined above.
• For some other reason, we will have some verified pieces of alignment technology, but AI employers won't use that technology by default
• Maybe because tech companies are much more reckless or near-sighted than I'm imagining?
• Governments are much more competent than I currently believe, or will become much more competent before the endgame.
• EAs are planning to go into policy to try to make the governmental reaction smaller and saner, rather than try to push the government into positive initiatives, and the EAs are well-coordinated about this.
• In a local takeoff scenario, the leading team is not concerned about alignment or is basically not cosmopolitan in its values.