Everything you said sounds correct, except the last bit, which is just unclear to me. I'd welcome a demonstration (or formal definition) some day:

By the way, you say "partial histories of sense data and actions". I try considering this problem in time-reversible dynamic, it adds a lot of elegance, and there actions are not part of history, but more like something that is removed from history. State of the agent doesn't accumulate from actions and observations, instead it's added to by observations and taken away from by actions. The point at which something is considered observation or action and not part of agent's state is itself rather arbitrary, and both can be seen as points of shifting the scope on what is considered part of agent. (This doesn't have anything agent-specific, and is more about processes in general.)