How does value scale with brain size? For example, if we had enough atoms to build 10 human-like brains, is it better to do that or to build a giant brain that can have much more complex thoughts and experiences? Are there experiences that we can't imagine today that are many times more valuable than what we can imagine, without even using more atoms? Also consider questions like population ethics and egoism vs altruism, e.g., should we use the rest of the universe to create more lives (if so how many), or to extend/improve the lives of existing people?
For something to count as a utopia for me, it would have to allow me to become smarter, answer these questions, then have the answers make a meaningful difference in the world. Otherwise, it could still be an AI disaster relative to the potential of the universe.
Good points and I agree with pretty much all of them, but for the sake of argument I'll try to write the strongest response I can:
It seems to me that your view of value is a little bit mystical. Our minds can only estimate the value of situations that are close to normal. There's no unique way to extend a messy function from [0,1] to [-100,100]. I know you want to use philosophy to extend the domain, but I don't trust our philosophical abilities to do that, because whatever mechanism created them could only test them on normal situations. We already see di...
I think I've come up with a fun thought experiment about friendly AI. It's pretty obvious in retrospect, but I haven't seen it posted before.
When thinking about what friendly AI should do, one big source of difficulty is that the inputs are supposed to be human intuitions, based on our coarse-grained and confused world models. While the AI's actions are supposed to be fine-grained actions based on the true nature of the universe, which can turn out very weird. That leads to a messy problem of translating preferences from one domain to another, which crops up everywhere in FAI thinking, Wei's comment and Eliezer's writeup are good places to start.
What I just realized is that you can handwave the problem away, by imagining a universe whose true nature agrees with human intuitions by fiat. Think of it as a coarse-grained virtual reality where everything is built from polygons and textures instead of atoms, and all interactions between objects are explicitly coded. It would contain player avatars, controlled by ordinary human brains sitting outside the simulation (so the simulation doesn't even need to support thought).
The FAI-relevant question is: How hard is it to describe a coarse-grained VR utopia that you would agree to live in?
If describing such a utopia is feasible at all, it involves thinking about only human-scale experiences, not physics or tech. So in theory we could hand it off to human philosophers or some other human-based procedure, thus dealing with "complexity of value" without much risk. Then we could launch a powerful AI aimed at rebuilding reality to match it (more concretely, making the world's conscious experiences match a specific coarse-grained VR utopia, without any extra hidden suffering). That's still a very hard task, because it requires solving decision theory and the problem of consciousness, but it seems more manageable than solving friendliness completely. The resulting world would be suboptimal in many ways, e.g. it wouldn't have much room for science or self-modification, but it might be enough to avert AI disaster (!)
I'm not proposing this as a plan for FAI, because we can probably come up with something better. But what do you think of it as a thought experiment? Is it a useful way to split up the problem, separating the complexity of human values from the complexity of non-human nature?