All of Nicolas Macé's Comments + Replies

[Apologies for the delay]

Is there necessarily a utility function for the predictor such that the predictor's behavior (which is arbitrary) corresponds to maximizing its own utility

You're right, the predictor's behavior might not be compatible with utility maximization against any beliefs. I guess we're often interested in cases where we can think of the predictor as an agent. The predictor's behavior might be irrational in the restrictive above sense,[1] but to the extent that we think of it as an agent, my guess is that we can still get away with usi... (read more)

I'd say that the connection is: Single-agent problems with predictors can be interpreted as sequential two-player games where the (perfect) predictor is a player who observes the action of the decision-maker and best-responds to it. In game-theoretic jargon, the predictor is a Stackelberg follower, and the decision-maker is the Stackelberg leader. (Related: (Kovarik, Oesterheld & Conitzer 2023))

2justinpombrio
What's the utility function of the predictor? Is there necessarily a utility function for the predictor such that the predictor's behavior (which is arbitrary) corresponds to maximizing its own utility? (Perhaps this is mentioned in the paper, which I'll look at.) EDIT: do you mean to reduce a 2-player game to a single-agent decision problem, instead of vice-versa?

Looks very interesting, I'll make sure to check out the git repo! Thanks for developing that!

As you're perhaps already aware, (Everitt, Leike & Hutter 2015) comes with a jupyter notebook that implements EDT and CDT in sequential decision problems. Perhaps useful as a comparison or a source of inspiration.

My view of decision theory now is that it's all about fixpoints. You solve some big equation, and inside of it is the same equation, and there are multiple fixpoint solutions, and you pick the (hopefully unique) best one.

Would you say that this is simi... (read more)

2justinpombrio
I was not aware of Everitt, Leike & Hutter 2015, thank you for the reference! I only delved into decision theory a few weeks ago, so I haven't read that much yet. Nash equilibria come from the fact that your action depends on your opponent's action, which depends on your action. When you assume that each player will greedily change their action if it improves their utility, the Nash equilibria are the fixpoints at which no player changes their action. In single-agent decision theory problems, your (best) action depends on the situation you're in, which depends on what someone predicted your action would be, which (effectively) depends on your action. If there's a deeper connection than this, I don't know it. There's a fundamental difference between the two cases, I think, because a Nash equilibrium involves multiple agents that don't know each others' decision process (problem statement: maximize the outputs of two functions independently), while single-agent decision theory involves just one agent (problem statement: maximize the output of one function).

Note for completeness:  Arif Ahmed uses a similar connection between Calvinism and evidential decision theory to introduce Newcomb's problem in Evidence, Decision and Causality.

3Jacob G-W
Thanks, I did not know this!

Seems like the payoffs of the two agents were swapped in the figure; this should be fixed now. Thanks for pointing it out!

I think the idea of contracts is interesting. I’m probably less optimistic (or pessimistic?) than the authors that the sub-agents can always contract to complete their preferences. For one thing, contracts might be expensive to make. Second, even free contracts might not be incentivized, for the usual reasons rational agents cannot always avoid inefficiencies in trading (cf the Myerson-Satterthwaite theorem).

On might object that Myerson–Satterthwaite doesn't allow for smart contracts that can conditionally disclose private information. But then I'd argue t... (read more)

One observation that dissolves a little bit of the mystery for me: We can devise, within ZF, a set of properties that points to a unique integer . We can write down this integer, of course, but ZF won't be able to notice that it's the one that we're after, because if it did it'd prove its own consistency.

Here's one of the math facts I find the hardest to wrap my head around: There are some finite, well-defined integers that are impossible to compute within PA or ZF.

The argument goes roughly like this: We'll define a finite integer that is such that computing would be equivalent to proving ZF's consistency. Then we're done because by an incompleteness theorem, ZF cannot prove its own consistency. So how to construct such a number? First construct a Turing machine that halts (on a blank tape) iff ZF is inconsistent. Say it has states. The busy beaver ... (read more)

3Nicolas Macé
One observation that dissolves a little bit of the mystery for me: We can devise, within ZF, a set of properties that points to a unique integer n. We can write down this integer, of course, but ZF won't be able to notice that it's the one that we're after, because if it did it'd prove its own consistency.
2Vladimir_Nesov
Things like this are not quite "particular integers" though in the informal sense (19 is centrally a "particular integer", BB(1000) isn't). They are more like integer-definitions whose meaning (as particular integers) can't be characterized in some ways. Similarly, the decision that an agent will eventually make is not a "particular decision" from the point of view of that agent, but a decision-definition whose meaning as a particular decision that agent can't yet divine, because it's not yet decided. Some external oracle might know more about the meaning of the decision-definition and predict what the decision will be before the agent makes the decision. And a stronger theory might "know" (in a given sense) which particular integer an integer-definition formulated in a weaker theory designates, or at least have some better characterization of it available.

Perhaps of interest that people in quantum many-body physics have related results. One keyword is "scrambling". Like in your case, they have a network of interacting units, and since interactions are local they have a lightcone outside of which correlations are exactly zero. 

They can say more than that: Because excitations typically propagate slower than the theoretical max speed (the speed of light or whatever thing is analogous) there's a region near the edge of the lightcone where correlations are almost zero. And then there's the bulk of correlati... (read more)