The Bayesian Agent
4royf
5Richard_Kennaway
6royf
3timtyler
3[anonymous]
2royf
4[anonymous]
2royf
3Kindly
0Shmi
0[anonymous]
0Shmi
0[anonymous]
0Shmi
1DaFranker
0Shmi
2Stuart_Armstrong
2NancyLebovitz
1Johnicholas
New Comment
If you're a devoted Bayesian, you probably know how to update on evidence, and even how to do so repeatedly on a sequence of observations. What you may not know is how to update in a changing world. Here's how:
%3d\Pr(W_{t+1}|O1,\ldots,O{t+1})%3d\frac{\sigma(O{t+1}|W{t+1})\cdot\Pr(W_{t+1}|O_1,\ldots,O_t)}{\sumw\sigma(O{t+1}|w)\cdot\Pr(w|O_1,\ldots,O_t)})
As usual with Bayes' theorem, we only need to calculate the numerator for different values of , and the denominator will normalize them to sum to 1, as probabilities do. We know
as part of the dynamics of the system, so we only need
). This can be calculated by introducing the other variables in the process:
%3d\sum_{W_t,A_t}\Pr(W_t,At,W{t+1}|O_1,\ldots,O_t))
An important thing to notice is that, given the observable history, the world state and the action
are independent - the agent can't act on unseen information. We continue:
\cdot\Pr(A_t|O_1,\ldots,Ot)\cdot%20p(W{t+1}|W_t,A_t))
Recall that the agent's belief is a function of the observable history, and that the action only depends on the observable history through its memory
. We conclude:
\cdot\pi(A_t|Bt)\cdot%20p(W{t+1}|W_t,A_t))
I'm not seeing how this lets the agent update itself. The formula requires knowledge of sigma, pi, and p. (BTW, could someone add to the comment help text instructions for embedding Latex?) pi is part of the agent but sigma and p are not. You say
We know sigma as part of the dynamics of the system
But all the agent knows, as you've described it so far, is the sequence of observations. In fact, it's stretching it to say that we know sigma or p -- we have just given these names to them. sigma is a complete description of how the world state determines what the agent senses, and p is a complete description of how the agent's actions affect the world. As the designer of the agent, will you be explicitly providing it with that information in some future instalment?
Everything you say is essentially true.
As the designer of the agent, will you be explicitly providing it with that information in some future instalment?
Technically, we don't need to provide the agent with p and sigma explicitly. We use these parameters when we build the agent's memory update scheme, but the agent is not necessarily "aware" of the values of the parameters from inside the algorithm.
Let's take for example an autonomous rover on Mars. The gravity on Mars is known at the time of design, so the rover's software, and even hardware, is built to operate under these dynamics. The wind velocity at the time and place of landing, on the other hand, is unknown. The rover may need to take measurements to determine this parameter, and encode it in its memory, before it can take it into account in choosing further actions.
But if we are thoroughly Bayesian, then something is known about the wind prior to experience. Is it likely to change every 5 minutes or can the rover wait longer before measuring again? What should be the operational range of the instruments? And so on. In this case we would include this prior in p, while the actual wind velocity is instead hidden in the world state (only to be observed occasionally and partially).
Ultimately, we could include all of physics in our belief - there's always some Einstein to tell us that Newtonian physics is wrong. The problem is that a large belief space makes learning harder. This is why most humans struggle with intuitive understanding of relativity or quantum mechanics - our brains are not made to represent this part of the belief space.
This is also why reinforcement learning gives special treatment to the case where there are unknown but unchanging parameters of the world dynamics: the "unknown" part makes the belief space large enough to make special algorithms necessary, while the "unchanging" part makes these algorithms possible.
For LaTeX instructions, click "Show help" and then "More Help" (or go here).
So the Bayesian belief is an expression for precisely everything the agent can possibly know about the world. Why not have the agent's memory represent exactly that?
The usual reasons for a lossless compressor not compressing maximally are that it is expensive and slow to compress - and for many applications you need rapid random access to the data.
two extremes...Bayesian belief
It is perfectly legal under the bayes to learn nothing from your observations. Or learn in the wrong direction, or sideways, or whatever. All depending on prior and inductive bias. There is no unique "Bayesian belief". If you had the "right" prior, you would find that would have to do very little updating, because the right prior is already right.
It is perfectly legal under the bayes to learn nothing from your observations.
Right, in degenerate cases, when there's nothing to be learned, the two extremes of learning nothing and everything coincide.
Or learn in the wrong direction, or sideways, or whatever.
To the extent that I understand your navigational metaphor, I disagree with this statement. Would you kindly explain?
There is no unique "Bayesian belief".
If you mean to say that there's no unique justifiable prior, I agree. The prior in our setting is basically what you assume you know about the dynamics of the system - see my reply to RichardKennaway.
However, given that prior and the agent's observations, there is a unique Bayesian belief, the one I defined above. That's pretty much the whole point of Bayesianism, the existence of a subjectively objective probability.
If you had the "right" prior, you would find that would have to do very little updating, because the right prior is already right.
This is true in a constant world, or with regard to parts of the world which are constant. And mind you, it's true only with high probability: there's always the slight chance that the sky is not, after all, blue.
But in a changing world, where part of the change is revealed to you through new observations, you have to keep pace. The right prior was right yesterday, today there's new stuff to know.
Right, in degenerate cases, when there's nothing to be learned, the two extremes of learning nothing and everything coincide.
In the case where your prior says "the past is not informative about the future". You learn nothing. A degenerate prior, not degenerate situation.
To the extent that I understand your navigational metaphor, I disagree with this statement. Would you kindly explain?
Imagine a bowl of jellybeans. you put in ten red and ten white. You take out 3, all of which are red, the probability of getting a red on the next draw is 7/17.
Take another boal, have a monkey toss in red beans and white beans with 50% probability. You draw 3 red, the draw probability is now 50% (becuase you had a maxentropy prior).
Take another boal. Beans were loaded in with unknown probabilitities. You draw 3 red, your draw probability is 4/5 red.
See how depening on your assumptions, you learn in different directions with the same observations? Hence you can learn in the wrong direction with a bad prior.
Learning sideways is a bit of metaphor-stretching, but if you like you can imagine observing 3 red beans proves the existence of god under some prior.
given that prior and the agent's observations
Yes yes. I was being pedantic because your post didn't talk about priors and inductive bias.
very little
where part of the change is revealed to you through new observations, you have to keep pace.
I thought of that. I didn't think enough. "very little" was the wrong phrasing. It's not that you do less updating, it's that your updates are on concrete things like "who took the cookies" instead of "does gravity go as the squre or the cube" because your prior already encodes correct physics. Very little updating on physics.
Imagine a bowl of jellybeans. [...]
Allow me to suggest a simpler thought experiment, that hopefully captures the essence of yours, and shows why your interpretation (of the correct math) is incorrect.
There are 100 recording studios, each recording each day with probability 0.5. Everybody knows that.
There's a red light outside each studio to signal that a session is taking place that day, except for one rogue studio, where the signal is reversed, being off when there's a session and on when there isn't. Only persons B and C know that.
A, B and C are standing at the door of a studio, but only C knows that it's the rogue one. How do their beliefs that there's a session inside change by observing that the red light is on? A keeps the 50-50. B now thinks it's 99-1. Only C knows that there's no session.
So your interpretation, as I understand it, would be to say that A and B updated in the "wrong direction". But wait! I practically gave you the same prior information that C has - of course you agree with her! Let's rewrite the last paragraph:
A, B and C are standing at the door of a studio. For some obscure reason, C secretly believes that it's the rogue one. Wouldn't you now agree with B?
And now I can do the same for A, by not revealing to you, the reader, the significance of the red lights. My point is that as long as someone runs a Bayesian update, you can't call that the "wrong direction". Maybe they now believe in things that you judge less likely, based on the information that you have, but that doesn't make you right and them wrong. Reality makes them right or wrong, unfortunately there's no one around who knows reality in any other way than through their subjective information-revealing observations.
No matter what, someone is still updating in the wrong direction, even if we don't know who it is.
Yes, the Newtonian force a mass exerts on another mass far away from the first one falls off as the square of the distance. It's the word "cube" that confuses me.
This doesn't establish when the information can actually be compressed to any sensible amount...
What you've got is probably complicated enough already, but (generalizing from one example) I think the way humans work is that the compressed memories aren't necessarily lost, they just get harder to access.
When it's necessary to make a major update, memories might get reexamined (and possibly remade, but perhaps not entirely so) and different features of the memories get high-lighted.
You might enjoy Crutchfield's epsilon machines, and Shalizi's CSSR algorithm for learning them:
http://masi.cscs.lsa.umich.edu/~crshalizi/notabene/computational-mechanics.html