LESSWRONG
LW

All of mushroomsoup's Comments + Replies

Instrumental Goals Are A Different And Friendlier Kind Of Thing Than Terminal Goals
mushroomsoup*30

This comment raises some good points, but even "there will be a natural pressure for [subprocesses] to resemble a corrigible agent" seem to be debatable. Again consider the restaurant setting. It is sometime necessary for restaurants to close temporarily for renovation to increase the seating capacity, upgrade equipment, etc. The head chef who decided to renovate will be making the instrumental goals of all the other chefs (make a good food, earn money to stay alive) untenable while they are furloughed. More generally, progress towards terminal goals is not monotonic and thus only focusing on the local topology of the optimization landscape might be insufficient to predict long-horizon trends. 

Reply
2Max Harms
This seems right. Some sub-properties of corrigibility, such as not subverting the higher-level and being shutdownable, should be expected in well-constructed sub-processes. But corrigibility is probably about more than just that (e.g. perhaps myopia) and we should be careful not to assume that well-constructed sub-processes that resemble agents will get all the corrigibility properties.
Orienting to 3 year AGI timelines
mushroomsoup10

Thanks! The linked article is exactly what I was looking for. Assuming "nationalization" means something like "soft nationalization" does make the timeline seem a lot more plausible. 

Reply
Orienting to 3 year AGI timelines
mushroomsoup60

According to the timeline of the post, AGI will take place during the Trump presidency and much of the nationalization efforts will need to be lead by his administration. However, that seems antithetical to the administrations general ethos of deregulation (at least in the banking and energy sectors). Would it be possible to explain which avenues would lead Trump to nationalize, for example, OpenAI or Antropic?

Reply
7Seth Herd
This is a good point. Nationalization is hard and complex, and it would probably slow progress - and the current administration would be against it on general principles, as you say. But I think people are underestimating the government's flexibility and willingness to exert control when things get weird and dangerous. Governments typically do just that. Even Soft Nationalization: How the US Government Will Control AI Labs underestimates this; perhaps this would happen in long timelines, but I think there are more direct but still easy routes to control when things heat up and the bright boys in national security realize what's going on. I expect a "softer nationalization" of the government just asking politely to be included in deliberations among org leadership. Existing emergency act procedures very likely apply as soon as you take AGIs security implications seriously. They don't have to nationalize in any strong sense to exert control over the technology. Anyone being asked politely by the NSA to do something they could legally demand would be wise to comply, or at least appear to comply.