(It follows that an artificial intelligence just a tiny bit smarter than Einstein and von Neumann would be as much more productive than them as they are in relation to other mathematician/physicists).
I don't think this necessarily does follow. I think it only follows that such an AI would be much more productive than average in a population of Einsteins.
I once heard an acquaintance of a friend of mine put it this way (back when I wasn't familiar with transhumanism and I didn't know whether he was wrong or right): “Life extension had better be called old age extension: you may die at 160 instead of 80, but it's not like when you're 40 you'll do the things people today do when they're 20.”
Anyway, going by revealed preferences, fewer people might dislike the former scenario than you might think.
I don't think that necessarily reveals people's preferences; that would imply that they choose that outcome. I think in most cases people are ignorant of what is going to happen, or know only in an abstract sense. Those who actually know what they're in for, tend not to die that way..
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
= 27d567bc2d48d9f40e9ccc20ea370b15)
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
We discussed this issue at the two MIRIx Boston workshops. A big problem with parliamentary models which we were unable to solve, was what we've been calling ensemble stability. The issue is this: suppose your AI's value system is made from a collection of value systems in a voting-like system, is constructing a successor, more powerful AI, and is considering constructing the successor so that it represents only a subset of the original value systems. Each value system which is represented will be in favor; each value system which is not represented, will be against. In order to keep that from happening, you either need a voting system which somehow reliably never does that (but nothing we tried worked), or a special case for constructing successors, and a working loophole-free definition of that case (which is Hard).
This seems to be almost equivalent to irreversibly forming a majority voting bloc. The only difference is how they interact with the (fake) randomization: by creating a subagent, it effectively (perfectly) correlates all the future random outputs. (In general, I think this will change the outcomes unless agents' (cardinal) preferences about different decisions are independent).
The randomization trick still potentially helps here: it would be in each representative's interest to agree not to vote for such proposals, prior to knowing which such proposals will come up and in which order they're voted on. However, depending on what fraction of its potential value an agent expects to be able to achieve through negotiations, I think that some agents would not sign such an agreement if they know they will have the chance to try to lock their opponents out before they might get locked out.
Actually, there seems to be a more general issue with ordering and incompatible combinations of choices - splitting that into a different comment.