Tim Tyler:

Yes, I understood that. I meant can you reduce it to non-mind? Besides, evolution is an optimization process, and I wouldn't call it an agent that has values.

Either way, the point is that it's just moving my confusion from the word "preferences" to the word "values".

If I then say "those things an agent wants", then I'm just stuck at the word "want" again. It's not a reduction, just a repeated renaming of the same black box.