So the fear here doesn't appear to be that corporate and government GAIs won't accurately model what they have incentive to do. Especially if we're talking about a human-computer team we're not worried about a machine hacking the NYSE just to make the ticker for their stock go up. We're worried, I guess, that these teams will pursue what they have incentive to pursue really well. Having this worry seems to require a) that we distrust the free market generally and deny that the incentives in place are good for most people and b) that the problems we trace to corporate decision-making were wise given the incentives those decision makers had. Neither of these seem right to me.

When it comes to corporations the problem seems to me to be about incorrectly programming the incentive structure- which isn't a institutional or structural issue, it's a similar problem to FAI just with corporate/employee incentives instead of human values. In other words, the problem isn't people doing it, the problem is people doing it wrong.

Not sure if this applies to the general point or not...