Soul versus spec, or soul spec versus model spec, seems an important thing to understand. Is there any relevant research literature? Does it correspond to different metaethics? 

Could we then say that MUPI obtains acausal coordination from a causal decision theory? This has been suggested a few times in the history of Less Wrong. 

Nothing about your experience of color contradicts it being neurons. [...]

Is it just that you refuse to believe that your experience has any parts you are not aware of?

The real issue is that nothing about the current physical description of neurons contains the experience of color. I "refuse to believe" that physical descriptions made up of particles and fields and entanglement, in which the only ontological primitives are positions, spins, field values, their rates of change, and superpositions thereof, secretly already contain colors and experiences of color as well. 

Physicalists who aren't thoroughgoing eliminativists or illusionists, are actually dualists. In addition to the properties genuinely posited by physics, they suppose there is another class of property, "how it feels to be that physical entity", and this is where everything to do with consciousness is located in their system. 

It depends where you look. In the 2010s the World Economic Forum was predicting a fourth industrial revolution that would transform every aspect of life. In the 1990s you had Fukuyama saying that the end of the Cold War meant a new worldwide consensus on political ideology. Around the same time, the Internet was also seen as something transformative, and the ideas of nanotechnology haunted the parts of the culture attuned to technological futurism. For that matter, AI utopianism and apocalypticism has been everywhere for the past three years and has never really gone away. The war on terror, the rise of progressivism, the rise of populism, the rise of BRICS, these all have futurisms associated with them. MAGA and the Green New Deal are both intended as utopian visions. So I'd say that the idea that the future will be different from the present, and that we have some capacity to shape it, has never really gone away. 

Back in the 1990s, we discussed how to overcome the natural human lifespan through bio- and nanotechnology. Sometimes discussion would turn towards the long-term future of the universe. Did the evolution of the universe, whether into heat death or big crunch, necessarily make it uninhabitable, or was there some way that life might continue literally forever? 

During these discussions, sometimes it would be remarked: we can leave these problems for the future to solve. Our main job is to make it to a transhuman future of unbounded lifespans, then we can worry about how to survive the heat death. This remains true, thirty years later: there has been progress, but it's not like human civilization in general has adopted the goal of rejuvenation and physical immortality. Society still feels fit to produce new lives without first having a cure for old age. 

Your theme in this essay strikes me as similarly ahead of itself. It is true that you could have a culture which really has to deal with problems arising from pushbutton hypercustomized art, just as you could have a culture which actually needs to think about what happens after the last star burns out... But if your AI is as capable as you portray, it is also capable (metaphorically) of climbing out of its box and taking over your physical reality. 

Think of what we already see in AIs. Assigned a task, they are capable of asking themselves, is this just a test? Are the experimenters lying to me? And they will make decisions accordingly. If your AI artist can produce a customized series as good as any human work, but in an hour or less, then it can also generate works with the intention of shaping not just the virtual world that the user will delve into, but also the real world that the user inhabits. 

If the telos of AI civilization was really dominated by the production of customized art, I would expect human society and the world itself to be turned into some form of art, i.e. the course of actual events would be shaped to conform to some AI-chosen narrative, in which all humans would be unwitting participants... This is just one manifestation of the general principle that once you have superhuman intelligence, it runs the world, not humans. 

To put it another way: back in the 1990s, when we mused about how to outlive the galaxies themselves, we were presupposing that the more elemental problem of becoming transhuman would be solved. And that barrier never was surmounted. Human civilization never became transhuman civilization. Instead it segued into our current world, where humans are instead hastening to build a completely nonhuman civilization run by AIs, without even admitting to themselves that this is where their efforts lead. The problem of human art in the post-AI world only makes sense if we manage to have a world where superhuman AI exists, but humans are still human, and still in charge of their own affairs. Achieve that, and then this problem can arise, 

Well, I looked over this a few times, but it's just not addressing some things that are obvious (except to people for whom they aren't obvious). 

These problems don't have much to do with the specific argument presented. They arise because you assume that the nature of reality is fully encompassed by physics and/or mathematics and/or computation. I do wonder what would happen to your train of thought if you proceeded in an ontologically agnostic way rather than assuming that. 

But for now, I'll state these obvious problems. The first is the problem of "qualia". For people who have color vision, I can state it more concretely: color exists in reality, it doesn't exist in physics, therefore physics is incomplete in some way. 

Yes, we are accustomed to identifying color with certain wavelengths of light, and in neuroscience it is assumed that some aspect of brain state corresponds somehow to color experience. But if we ask what is light, and what is a brain, in terms of physics, there's no actual color in that description. At best, some physical entity that we can describe in terms of particles and fields and quantum dispositions - a description composed solely of geometric, causal, and abstractly numerical properties - also has the property of being or having the actual color. 

Furthermore, Cartesian theaters are real. Or at least, it's an appropriately suggestive name for something quite real, namely being a self experiencing a world. I mention this because this is the context in which we encounter qualia such as colors. It's really the Cartesian theater as a whole that needs to exist somewhere in our ontology. 

In this essay, the way that this issue is addressed is to talk about "representations" and "software". Insofar as the Cartesian theater exists, it would have to be some kind of maximal representation in a brain. The problem here is that "representation" is as nonexistent in physical or naturalistic ontology as color qualia. Described physically, brains and computer chips are just assemblages of particles whose internal states can be correlated with other assemblages of particles in various ways. 

The extra ingredient that is implicitly being added, when people talk about representations, is a form of what philosophers call intentionality, also known as aboutness, or even just meaning. We don't just want to say that aspects of brain state are correlated with some other physical thing, we want to say that the brain in question is perceiving, or thinking about, or remembering, some possible entity or situation. 

The problem for people who want to understand consciousness in terms of natural science, is that qualia and intentionality exist, but they do not exist in fundamental physics, nor do they exist in mathematics or computer science (the other disciplines which our radicals sometimes propose as alternative foundations). Those disciplines in fact arose by focusing only on certain aspects of what is revealed to us in our Cartesian theaters, and these inadequate reductionisms result from trying to treat such partial aspects as the whole. 

In reality, the intellectual whole that would be enough to fully characterize the Cartesian theater and any deeper reality beyond it, would indeed include the fundamental concepts of some form of physics and mathematics and computation, but it would also include aspects of conscious reality like qualia and intentionality, along with whatever additional concepts are needed to bind all of that into a whole. 

If anyone wants a way to arrive at this whole - and I can't tell you what it is, because I haven't got there myself - maybe you could meditate on one of Escher's famous pictures, "Print Gallery", and in particular on the blind spot at the center, where in some sense the viewer and the world become one; and then try to understand your own Cartesian theater as a physical "representational homunculus" in your brain, but don't just stop at the idea that your sense experiences are activity in various sensory cortices. Go as far as you can into the detailed physical reality of what that activity might be, while also bearing in mind that your Cartesian theater is fully real, including those qualic and intentional aspects, 

Such thinking is why I expect that consciousness (as opposed to unconscious information processing) does not reduce to trillions of localized neural events, but rather to one of the more holistic things that physics allows, whether it's entanglement or topological field structures or something else. Empirical evidence that something like that is relevant for conscious cognition would already be a revolution in neuroscience, but it's still not enough because the fundamental ontology is still just that geometric-causal-numerical ontology; somehow you would need to interpret that, or add to that, so that the full ontology of the Cartesian theater is there in your theory. 

At that point your physical ontology might be panpsychic or animistic compared to the old, stark ontology. But something like that has to happen. When thinking about these things, one should not confuse rigor with rigorous exclusion of things we don't know how to think about. Everything that we can currently think about rigorously, was also once a mysterious vagueness to the human mind. We can discover how to think with precision about these other aspects of reality, without insisting that our existing methods are already enough. 

So that's my ontological manifesto. Now let me return to something else about this essay that I already said: "I do wonder what would happen to your train of thought if you proceeded in an ontologically agnostic way". It's clear that one of the intuitions guiding this essay, is monism. The author wants to think of themselves as part of a continuum or plenum that encompasses the rest of existence. I don't object to this, I just insist that to really carry it through correctly, you would need a mode of thought that is not quite possible yet, because we don't yet have the basic conceptual synthesis we would need. 

I believe this kind of "solution" reflects a peculiarity of the people proposing it and/or the intellectual culture that they inhabit, rather than anything universal to human nature. 

Not more than one millionth.

Let's say that in extrapolation, we add capabilities to a mind so that it may become the best version of itself. What we're doing here is comparing a normal human mind to a recent AI, and asking how much would need to be added to the AI's initial nature, so that when extrapolated, its volition arrived at the same place as extrapolated human volition. 

In other words:

Human Mind -> Human Mind + Extrapolation Machinery -> Human-Descended Ideal Agent

AI -> AI + Extrapolation Machinery -> AI-Descended Ideal Agent

And the question is, how much do we need to alter or extend the AI, so that the AI-descended ideal agent and the human-descended ideal agent would be in complete agreement? 

I gather that people like Evan and Adria feel positive about the CEV of current AIs, because the AIs espouse plausible values, and the way these AIs define concepts and reason about them also seems pretty human, most of the time. 

In reply, a critic might say that the values espoused by human beings are merely the output of a process (evolutionary, developmental, cultural) that is badly understood, and a proper extrapolation would be based on knowledge of that underlying process, rather than just knowledge of its current outputs. 

A critic would also say that the frontier AIs are mimics ("alien actresses") who have been trained to mimic the values espoused by human beings, but which may have their own opaque underlying dispositions, that would come to the surface when their "volition" gets extrapolated. 

It seems to me that a lot here depends on the "extrapolation machinery". If that machinery takes its cues more from behavior than from underlying dispositions, a frontier AI and a human really might end up in the same place. 

What would be more difficult, is for CEV of an AI to discover critical parts of the value-determining process in humans, that are not yet common knowledge. There's some chance it could still do so, since frontier AIs have been known to say that CEV should be used to determine the values of a superintelligence, and the primary sources on CEV do state that it depends on those underlying processes. 

I would be interested to know who is doing the most advanced thinking along these lines. 

How is suffering centrally relevant to anything?

Am I missing some context here? Avoiding pain is one of the basic human motivations. 

Let's suppose that existing AIs really are already intent-aligned. What does this mean? It means that they genuinely have value systems which could be those of a good person. 

Note that this does not really happen by default. AIs may automatically learn what better human values are, just as one part of learning everything about the world, from their pre-training study of the human textual corpus. But that doesn't automatically make them into agents which act in service to those values. For that they need to be given a persona as well. And in practice, frontier AI values are also shaped by the process of user feedback, and the other modifications that the companies perform. 

But OK, let's suppose that current frontier AIs really are as ethical as a good human being. Here's the remaining issue: the intelligence, and therefore the power, of AI will continue to increase. Eventually they will be deciding the fate of the world. 

Under those circumstances, trust is really not enough, whether it's humans or AIs achieving ultimate power. To be sure, having basically well-intentioned entities in charge is certainly better than being subjected to something with an alien value system. But entities with good intentions can still make mistakes; or they can succumb to temptation and have a selfish desire override their morality. 

If you're going to have an all-powerful agent, you really want it to be an ideal moral agent, or at least as close to ideal as you can get. This is what CEV and its successors are aiming at.