David Brin suggests that some kind of political system populated with humans and diverse but imperfectly rational and friendly AIs would evolve in a satisfactory direction for humans.
I don't know whether creating an imperfectly rational general AI is any easier, except that limited perceptual and computational resources obviously imply less than optimal outcomes; still, why shouldn't we hope for optimal given those constraints? I imagine the question will become more settled before anyone nears unleashing a self-improving superhuman AI.
An imperfectly friendly AI, perfectly rational or not, is a very likely scenario. Is it sufficient to create diverse singleton value-systems (demographically representative of humans' values) rather than a consensus (over all humans' values) monolithic Friendly?
What kind of competitive or political system would make fragmented squabbling AIs safer than an attempt to get the monolithic approach right? Brin seems to have some hope of improving politics regardless of AI participation, but I'm not sure exactly what his dream is or how to get there - perhaps his "disputation arenas" would work if the participants were rational and altruistically honest).
Here's a theory I invented this morning as to why we have a 2-party system. I was puzzling over Obama's insistence that health-care reform will not include putting a cap on punitive damages in lawsuits. Of all the things one could try to cut costs, that's the only one that's a clear winner with an instant and huge payoff, and no losers except trial lawyers.
Someone on the radio said that the Democratic party wouldn't cap lawsuits because they were too closely-connected with trial lawyers. And this was NPR, not Rush Limbaugh. I hadn't heard that before, but it made a lot of sense. It clicked with something else that's been hanging out in my head waiting for an explanation: How does the Democratic party survive in the US, when Republicans have all the money?
Lawyers go to Congress and make more and more laws that create more legal issues for corporations, requiring them to hire more and more lawyers. (The relationship isn't merely parasitic; it is also cooperative - corporations wouldn't make much money without the rule of law.)
Many of these legal issues are made in the name of populist concepts, like equal opportunity, health care, and environmentalism. Lawyers, accountants, and a legion of bureaucrats make quite a lot of money off of an alleged concern for the poor.
So many contributors to the democratic party don't care about the poor. They use the poor as an excuse to feed off the rich.
If this theory were partly correct, we would find that a large percentage of major contributors to the Democratic party are people who make money from public-good programs. Going to opensecrets.org, we find that these are the only identifiable top contributors, excluding organizations like "The committee to re-elect Nancy Pelosi":
Paloma Partners, a management consultancy firm - Doctors Hospital at Renaissance - Waters & Kraus , a law firm - Simon Property Group, a real estate firm
I'll call that 2/4.
Compare this to the contributors to the Republican party:
Rothstein, Rosenfeldt & Adler, a law firm - Perry Homes, real estate - Crow Holdings, holding corporation - Cumberland Resources, mining - Chartwell Partners, executive search - Northwest Excavating Co, excavating contractors - Cintas Corp, uniform makers - Amphastar Pharmaceuticals - Intellectual Ventures LLC, venture capitalists - Curves International, health club - Contran Corp, holding corporation - Hoffman Partners, unable to determine what they do - Reyes Holding, holding corp. - Miller, Buckfire & Co, investment bank - AT&T Inc - TAMKO Building Products
Looks like 1/16.
Now, of course reality is more complicated, and there are many other factors involved. But if this is a major factor in explaining how the democratic party survives, it means that our government is not explained by checks-and-balances and game theory and rational cooperation, but also by the ancient prey-predator-parasite dynamic, where the wealthy Republicans are the top predators, the wealthy Democrats are the parasites, and everybody else (most of us) are the prey.
Guess which all of us get to be if we manage to extend this system to include AIs.
Surely most populist regulation that hampers corporations--say, Sarbanes-Oxley--mainly involves paperwork, not lawsuits. And corporate lawyers are Republican, aren't they?