In AI research, intelligent agents typically have a clear-cut and well-defined final goal, e.g., win the chess game or drive the car to the destination legally. The same holds for most tasks that we assign to humans, because the time horizon and context is known and limited. (...) a truly well-defined goal would specify how all particles in our Universe should be arranged at the end of time.
We typically only care about the arrangement of particles at the end of the task, because that is the nature of the simple tasks we usually use machines for today. Actually, even that is not true: when "driving the car to destination legally" we care not only about the arrangement of the particles of the car at the end of the trip, but also about what happened on the way -- that's what "legally" means here. (Unless we also count "police sending us tickets" as particles. But I guess the car is supposed to follow the laws even when the police does not look.)
We can define "journey" goals e.g. by calculating score at each time interval, and trying to maximize the sum or the average (or some other function) of all the intervals. This can make sense even if we don't know how long the task will last.
treat experience as inherently positive and not try to distinguish between positive and negative experiences.
This sounds wrong. But I am not even sure what exactly would we measure here, if both positive and negative experience count the same. Is it the intensity of the experience (in either direction) which counts? (That is, would you rather be tortured than bored? Would you rather be tortured really painfully than enjoying a mild pleasure?) Or is it duration of the experience? (That is, we want to maximize the subjective time of sentient beings, regardless of what happens during the time? Would you rather live 1001 years in hell than 1000 years in heaven?)
This sounds wrong.
Of course. That's why I proposed refining it.
But I am not even sure what exactly would we measure here
I thought it was obvious. It is the integral of total experience (suitably defined) through time that counts.
As Tegmark argues, the idea of "final goal" for AI is likely incoherent, at least if (as he states), "Quantum effects aside, a truly well-defined goal would specify how all particles in our Universe should be arranged at the end of time."
But "life is a journey not a destination". So really, what we should be specifying is the entire evolution of the universe through its lifespan. So how can the universe "enjoy itself" as much as possible before the big crunch (or before and during the heat death)*.
I hypothesize that experience is related to, if not a product of, change. I further propose (counter-intuitively, and with an eye towards "refinement" (to put it mildly))** that we treat experience as inherently positive and not try to distinguish between positive and negative experiences.
Then it seems to me the (still rather intractable) question is: how does the rate of entropy's increase relate to the quantity of experience produced? Is it simply linear (in which case, it doesn't matter, ethically)? My intuition is that is it more like the fuel efficiency of a car, non-linear and with a sweet spot somewhere between a lengthy boredom and a flash of intensity.
*I'm not super up on cosmology; are there other theories I ought to be considering?
**One idea for refinement: successful "prediction" (undefined here) creates positive experiences; frustrated expectations negative ones.