How similar are their values actually?
One obvious difference seems to be their position on the exploration/exploitation scale, most corporations do not get bored (the rare cases where they do seem to get bored can probably be explained by an individual executive getting bored, or by customers getting bored and the corporation managing to adapt).
Corporations also do not seem to have very much compassion for other corporations, while they do sometimes co-operate I have yet to see an example one corporation giving money to another, without anticipating some sort of gain from this action (any altruism they display towards humans is more likely caused by the individuals running things or done for signalling purposes, if they were really altruistic you would expect it to be towards each-other).
Do they really value independence and individuality? If so then why do they sometimes merge? I suppose you could say that the difference between they and humans is that they can merge while we can't, but I'm not convinced we would do so even if we could.
There may be superficial similarities between their values and ours, but it seems to me like we're quite different where it matters most. A hypothetical future which lacks creativity, altruism or individuality can be safely considered to have lost almost all of its potential value.
Altruism and merging: Two very good points!
Altruism can be produced via evolution by kin selection or group selection. I don't think kin selection can work for corporations, for several reasons, including massive lateral transfer of ideas between corporations (so that helping a kin does not give a great boost to your genes), deliberate acquisition of memes predominating over inheritance, and the fact that corporations can grow instead of reproducing, and so are unlikely to be in a position where they have no growth potential themselves but can help a kin instead.
Can group selection apply to corporations?
What are the right units of selection / inheritance?
Completely artificial intelligence is hard. But we've already got humans, and they're pretty smart - at least smart enough to serve some useful functions. So I was thinking about designs that would use humans as components - like Amazon's Mechanical Turk, but less homogenous. Architectures that would distribute parts of tasks among different people.
Would you be less afraid of an AI like that? Would it be any less likely to develop its own values, and goals that diverged widely from the goals of its constituent people?
Because you probably already are part of such an AI. We call them corporations.
Corporations today are not very good AI architectures - they're good at passing information down a hierarchy, but poor at passing it up, and even worse at adding up small correlations in the evaluations of their agents. In that way they resemble AI from the 1970s. But they may provide insight into the behavior of AIs. The values of their human components can't be changed arbitrarily, or even aligned with the values of the company, which gives them a large set of problems that AIs may not have. But despite being very different from humans in this important way, they end up acting similar to us.
Corporations develop values similar to human values. They value loyalty, alliances, status, resources, independence, and power. They compete with other corporations, and face the same problems people do in establishing trust, making and breaking alliances, weighing the present against the future, and game-theoretic strategies. They even went through stages of social development similar to those of people, starting out as cutthroat competitors, and developing different social structures for cooperation (oligarchy/guild, feudalism/keiretsu, voters/stockholders, criminal law/contract law). This despite having different physicality and different needs.
It suggests to me that human values don't depend on the hardware, and are not a matter of historical accident. They are a predictable, repeatable response to a competitive environment and a particular level of intelligence.
As corporations are larger than us, with more intellectual capacity than a person, and more complex laws governing their behavior, it should follow that the ethics developed to govern corporations are more complex than the ethics that govern human interactions, and a good guide for the initial trajectory of values that (other) AIs will have. But it should also follow that these ethics are too complex for us to perceive.