= 27d567bc2d48d9f40e9ccc20ea370b15)
Anja comments on Universal agents and utility functions - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (38)
This seems unnecessary. The information u_i is already contained in x_i.
This completely breaks the expectimax principle. I assume you actually mean something like=\textrm{arg}\max_{y_k}\sum_{x_k}u_k(\.{y}\.{x}_{%3Ck}y\underline{x}_{k:n})M(\.{y}\.{x}_{%3Ck}y\underline{x}_{k:n}))
which is just Agent 2 in disguise.
Oops. Yes, that's what I meant. But it is not the same as Agent 2, because this (Agent 4?) uses its current utility function to evaluate the desirability of future observations and actions, even though it knows that it will use a different utility function to choose between them later. For example, Agent 4 will not take the Simpleton's Gambit because it cares about its current utility function getting satisfied in the future, not about its future utility function getting satisfied in the future.
Agent 4 can be seen as a set of agents, one for each possible utility function, that are using game theory with each other.
I second the general sentiment that it would be good for an agent to have these traits, but if I follow your equations I end up with Agent 2.
No, you don't. If you tried to represent Agent 2 in that notation, you would get
modeled_action(n, k) = argmax(y_k) sum(x_k) [u_k(yx_<k, yx_k) + u_(k+1)(yx_<k, yx_k:k+1) + ... + u_n(yx_<k, yx_k:n)]*M(yx_<k, yx_k:n), where y_m = modeled_action(n, m) for m>k.
You were using u_k to represent the utility of the last step of its input, so that total utility is the sum of the utilities of its prefixes, while I was using u_k to represent the utility of the whole sequence. If I adapt Agent 4 to your use of u_k, I get
modeled_action(n, k) = argmax(y_k) sum(x_k) [u_k(yx_<k, yx_k) + u_k(yx_<k, yx_k:k+1) + ... + u_k(yx_<k, yx_k:n)]*M(yx_<k, yx_k:n), where y_m = modeled_action(n, m) for m>k.
I am starting to see what you mean. Let's stick with utility functions over histories of length m_k (whole sequences) like you proposed and denote them with a capital U to distinguish them from the prefix utilities. I think your Agent 4 runs into the following problem: modeled_action(n,m) actually depends on the actions and observations yx_{k:m-1} and needs to be calculated for each combination, so y_m is actually
which clutters up the notation so much that I don't want to write it down anymore.
We also get into trouble with taking the expectation, the observations x_{k+1:n} are only considered in modeling the actions of the future agents, but not now. What is M(yx_<k,yx_k:n) even supposed to mean, where do the x's come from?
So let's torture some indices:
where n>=k and
This is not really AIXI anymore and I am not sure what to do with it, but I like it.
Yes.
Oops, you are right. The sum should have been over x_{k:n}, not just over x_k.
Yes, that is a cleaner and actually correct version what I was trying to describe. Thanks.