I think "agent" or "agency" is being used in two different senses here — a descriptive/game-theoretical sense and a normative/political sense.

In the game-theory sense of "agent", noticing the presence of an "agent" does not imply "you'd want to respect their desires". For instance, Clippy is an agent, but an adversarial one. We don't want Clippy to get what it wants with our light-cone, thank you very much.

The normative/political sense of "agency" implies a whole slew of values and norms having to do with how humans ought to relate to each other: people ought to get to make their own decisions; others ought not conspire to keep them ignorant; and so on.