I don't know of a game-theoretic formalism allowing for agents to win something other than generic "dollars" or "points", such that we can encode in the formalism that agents share some values but not others, and have tradeoffs among their different values.

I suspect this isn't the main obstacle to reducing ethics to game theory. Once I'm willing to represent agents' preferences with utility functions in the first place, I can operationalize "agents share some values" as some features of the world contributing positively to the utility functions of multiple agents, while an agent having "tradeoffs among their different values" is encoded in the same way as any other tradeoff they face between two things — as a ratio of marginal utilities arising from a marginal change in either of the two things.