= 27d567bc2d48d9f40e9ccc20ea370b15)
AlexMennen comments on Universal agents and utility functions - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (38)
No, you don't. If you tried to represent Agent 2 in that notation, you would get
modeled_action(n, k) = argmax(y_k) sum(x_k) [u_k(yx_<k, yx_k) + u_(k+1)(yx_<k, yx_k:k+1) + ... + u_n(yx_<k, yx_k:n)]*M(yx_<k, yx_k:n), where y_m = modeled_action(n, m) for m>k.
You were using u_k to represent the utility of the last step of its input, so that total utility is the sum of the utilities of its prefixes, while I was using u_k to represent the utility of the whole sequence. If I adapt Agent 4 to your use of u_k, I get
modeled_action(n, k) = argmax(y_k) sum(x_k) [u_k(yx_<k, yx_k) + u_k(yx_<k, yx_k:k+1) + ... + u_k(yx_<k, yx_k:n)]*M(yx_<k, yx_k:n), where y_m = modeled_action(n, m) for m>k.
I am starting to see what you mean. Let's stick with utility functions over histories of length m_k (whole sequences) like you proposed and denote them with a capital U to distinguish them from the prefix utilities. I think your Agent 4 runs into the following problem: modeled_action(n,m) actually depends on the actions and observations yx_{k:m-1} and needs to be calculated for each combination, so y_m is actually
which clutters up the notation so much that I don't want to write it down anymore.
We also get into trouble with taking the expectation, the observations x_{k+1:n} are only considered in modeling the actions of the future agents, but not now. What is M(yx_<k,yx_k:n) even supposed to mean, where do the x's come from?
So let's torture some indices:
where n>=k and
This is not really AIXI anymore and I am not sure what to do with it, but I like it.
Yes.
Oops, you are right. The sum should have been over x_{k:n}, not just over x_k.
Yes, that is a cleaner and actually correct version what I was trying to describe. Thanks.