What the agent tries to infer, is predictor's behavior for each of agent's possible actions, not just unconditional predictor's behavior. Being able to infer predictor's decision without assuming agent's action is equivalent to conclusing that predictor's decision is a constant function of agent's action (in agent's opinion, given the kind of decision-maker our agent is, which is something that it should be able to control better, but current version of the theory doesn't support).