But how does an agent introduce its values in the mixture?
I don't accept Will's overall position or reasoning but this particular part is relatively straightforward. It's just the same as how anyone negotiates. In this case the negotiation is just a little... indirect. (Expanded below.)
The agent is the way it decides, so at least in one interpretation its values must be reflected in its decisions (reasons for its decisions), seen in them, even if in a different interpretation its decisions reflect the mixed values of all things (for that is one thing the agent might want to take into account, as it becomes more capable of doing so).
An agent's decisions are determined by it's values but this relationship is many to one. For any given circumstances that an agent could be in all sorts of preferences will end up resolving to the same decision. If you decided to throw away all that information by only considering what can be inferred from the resultant decision then you will end up wrong. More importantly this isn't what the other agents will be doing so you will be wrong about them too.
Consider the coordination game as described by paulfchristiano, adopted as a metaphysics of morality by Will and that I'll consider as a counterfactual:
It is trivial* to see that this game reduces to equivalent to a simple two party prisoners dilemma with full mutual information. Each agent calculates the most efficient self interested bargains that could be made between them all and chooses to either act as if those bargains have been made or doesn't depending on whether it (reliably) predicts the other agents do likewise.
For all the agents when we look at their behavior we see them all acting equivalently to whatever the negotiated outcome comes out to. That tells us little about their individual values - we've thrown that information away and just elected to keep "Cooperate with negotiated preferences". But the individual preferences have been 'thrown into the mix' already back at the point where each of the agents considers the expected behavior of the others. (And there is no way that one of the agents will cooperate without it's values in the mix and all the agents like to win, etc, etc, and a lot more 'trivial'.)
I don't accept the premises here and so definitely don't accept any 'universal morality' but the "But how does an agent introduce its values in the mixture?" just isn't the weakpoint of the reasoning. It's tangent and to the extent that it is presented as objection it is a red herring.
* In the come back 20 minutes later and say "Oh, it's trivial" sense.
It is trivial* to see that this game reduces to equivalent to a simple two party prisoners dilemma with full mutual information.
It only reduces to/is equivalent to a prisoner's dilemma for certain utility functions (what you're calling "values"). The prisoners' dilemma is characterized by the fact that there is a dominant strategy equilibrium which is not Pareto optimal. But if the utility functions of the agents are such that the game is zero-sum, then this can't be the case, as every outcome is Pareto optimal in a zero-sum game.
Furthermore, ...
Here's the new thread for posting quotes, with the usual rules: