It may be that all sufficiently wise agents pursue the same goals because of decision theoretic considerations, by implicitly bargaining with each other and together pursuing some mixture of all of their values.
But how does an agent introduce its values in the mixture? The agent is the way it decides, so at least in one interpretation its values must be reflected in its decisions (reasons for its decisions), seen in them, even if in a different interpretation its decisions reflect the mixed values of all things (for that is one thing the agent might want to take into account, as it becomes more capable of doing so).
Why do I write this comment? I decided to do so, which tells something about the way I decide. Why do I write this comment? According to the laws of physics. There seems to be no interesting connection between such explanations, even though both of them hold, and there is peril in confusing them (for example, nihilist ethical ideas following form physical determinism).
But how does an agent introduce its values in the mixture?
I don't accept Will's overall position or reasoning but this particular part is relatively straightforward. It's just the same as how anyone negotiates. In this case the negotiation is just a little... indirect. (Expanded below.)
...The agent is the way it decides, so at least in one interpretation its values must be reflected in its decisions (reasons for its decisions), seen in them, even if in a different interpretation its decisions reflect the mixed values of all things (for that is one thing
Here's the new thread for posting quotes, with the usual rules: