= 27d567bc2d48d9f40e9ccc20ea370b15)
Tyrrell_McAllister comments on UDT agents as deontologists - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (109)
I have added a link (pdf) to a complete description of what a UDT algorithm is. I am confident that there are no "misinterpretations" there, but I would be grateful if you pointed out any that you perceive.
I believe it is an accurate description of UDT as presented in the original post, although incomplete knowledge about P_i can be accommodated without changing the formalism, by including all alternatives (completely described this time) enabled by available knowledge about the corresponding world programs, in the list {P_i} (which is the usual reading of "possible world"). Also note that in this post Wei Dai corrected the format of the decisions from individual input/output instances to global strategy-selection.
How important is it that the list {P_i} be finite? If P_i is one of the programs in our initial list that we're uncertain about, couldn't there be infinitely many alternative programs P_i1, P_i2, . . . behind whatever we know about P_i?
I was thinking that incomplete knowledge about the P_i could be captured (within the formalism) with the mathematical intuition function. (Though it would then make less sense to call it a specifically mathematical intuition.)
I've added a description of UDT1.1 to my pdf.
In principle, it doesn't matter, because you can represent a countable list of programs as a single program that takes an extra parameter (but then you'll need to be more careful about the notion of "execution histories"), and more generally you can just include all possible programs in the list and express the level to which you care about the specific programs in the way mathematical intuition ranks their probability and the way utility function ranks their possible semantics.
On execution histories: note that a program is a nice finite inductive definition of how that program behaves, while it's unclear what an "execution history" is, since it's an infinite object and so it needs to be somehow finitely described. Also, if, as in the example above you have the world program taking parameters (e.g. a universal machine that takes a Goedel number of a world program as parameter), you'll have different executions depending on parameter. But if you see a program as a set of axioms for a logical theory defining the program's behavior, then execution histories can just be different sets of axioms defining program's behavior in a different way. These different sets of axioms could describe the same theories, or different theories, and can include specific facts about what happens during program execution on so and so parameters. Equivalence of such theories will depend on what you assume about the agent (i.e. if you add different assumptions about the agent to the theories, you get different theories, and so different equivalences), which is what mathematical intuition is trying to estimate.
It's not accurate to describe strategies as mappings f: X->Y. A strategy can be interactive: it takes input, produces an output, and then environment can prepare another input depending on this output, and so on. Think normalization in lambda calculus. So, the agent's strategy is specified by a program, but generally speaking this program is untyped.
Let's assume that there is a single world program, as described here. Then, if A is the agent's program known to the agent, B is one possible strategy for that program, given in form of a program, X is the world program known to the agent, and Y is one of the possible world execution histories of X given that A behaves like B, again given in form of a program, then mathematical intuition M(B,Y) returns the probability that the statement (A~B => X~Y) is true, where A~B stands for "A behaves like B", and similarly for X and Y. (This taps into the ambient control analysis of decision theory.)
I'm following this paragraph from Wei Dai's post on UDT1.1:
So, "input/output mappings" is Wei Dai's language. Does he not mean mappings between the set of possible inputs and the set of possible outputs?
It seems to me that this could be captured by the right function f: X -> Y. The set I of input-output mappings could be a big collections of GLUTs. Why wouldn't that suffice for Wei Dai's purposes?
ETA: And it feels weird typing out "Wei Dai" in full all the time. But the name looks like it might be Asian to me, so I don't know which part is the surname and which is the given name.
I've been wondering why people keep using my full name around here. Yes, the name is Chinese, but since I live in the US I follow the given-name-first convention. Feel free to call me "Wei".
No, you can't represent an interactive strategy by a single input to output mapping. That post made a step in the right direction, but stopped short of victory :-). But I must admit, I forgot about that detail in the second post, so you've correctly rendered Wei's algorithm, although using untyped strategies would further improve on that.
Why not?
BTW, in UDT1.1 (as well as UDT1), "input" consists of the agent's entire memory of the past as well as its current perceptions. Thought I'd mention that in case there's a misunderstanding there.
... okay, this question allowed me to make a bit of progress. Taking as a starting point the setting of this comment (that we are estimating the probability of (A~B => X~Y) being true, where A and X are respectively agent's and environment's programs, B and Y programs representing agent's strategy and outcome for environment), and the observations made here and here, we get a scheme for local decision-making.
Instead of trying to decide the whole strategy, we can just decide the local action. Then, the agent program, and "input" consisting of observations and memories, together make up the description of where the agent is in the environment, and thus where its control will be applied. The action that the agent considers can then be local, just something the agent does at this very moment, and the alternatives for this action are alternative statements about the agent: thus, instead of considering a statement A~B for agent's program A and various whole strategies B, we consider just predicates like action1(A) and action2(A) which assert A to choose action 1 or action 2 in this particular situation, and which don't assert anything else about its behavior in other situations or on other counterfactuals. Taking into account other actions that the agent might have to make in the past or in the future happens automatically, because the agent works with complete description of environment, even if under severe logical uncertainty. Thus, decision-making happens "one bit at a time", and the agent's strategy mostly exists in the environment, not under in any way direct control by the agent, but still controlled in the same sense everything in the environment is.
Thus, in the simplest case of a binary local decision, mathematical intuition would only take as explicit argument a single bit, which indicates what assertion is being made about [agent's program together with memory and observations], and that is all. No maps, no untyped strategies.
This solution was unavailable to me when I thought about explicit control, because the agent has to coordinate with itself, rely on what it can in fact decide in other situations and not what it should optimally decide, but it's a natural step in the setting of ambient control, because the incorrect counterfactuals are completely banished out of consideration, and environment describes what the agent will actually do on other occasions.
Going back to the post explicit optimization of global strategy, the agent doesn't need to figure out the global strategy! Each of the agent copies is allowed to make the decision locally, while observing the other copy as part of the environment (in fact, it's the same problem as "general coordination problem" I described on the DT list, back when I was clueless about this approach).
Well, that was my approach in UDT1, but then I found a problem that UDT1 apparently can't solve, so I switched to optimizing over the global strategy (and named that UDT1.1).
Can you re-read explicit optimization of global strategy and let me know what you think about it now? What I called "logical correlation" (using Eliezer's terminology) seems to be what you call "ambient control". The point of that post was that it seems an insufficiently powerful tool for even two agents with the same preferences to solve the general coordination problem amongst themselves, if they only explicitly optimize the local decision and depend on "logical correlation"/"ambient control" to implicitly optimize the global strategy.
If you think there is some way to get around that problem, I'm eager to hear it.
So far as I can see, your mistake was assuming "symmetry", and dropping probabilities. There is no symmetry, only one of the possibilities is what will actually happen, and the other (which I'm back to believing since the last post on DT list) is inconsistent, though you are unlikely to be able to actually prove any such inconsistency. You can't say that since (S(1)=A => S(2)=B) therefore (S(1)=B => S(2)=A). One of the counterfactuals is inconsistent, so if S(1) is in fact A, then S(1)=B implies anything. But what you are dealing with are probabilities of these statements (which possibly means proof search schemes trying to prove these statements and making a certain number of elementary assumptions, the number that works as the length of programs in universal probability distribution). These probabilities will paint a picture of what you expect the other copy to do, depending on what you do, and this doesn't at all have to be symmetric.
...but on the other hand, you don't need the "input" at all, if decision-making is about figuring out the strategy. You can just have a strategy that produces the output, with no explicit input. The history of input can remain implicit in the agent's program, which is available anyway.
Good; that was my understanding.
Yes, that works too. On second thought, extracting output in this exact manner, while pushing everything else to the "input" allows to pose a problem specifically about the output in this particular situation, so as to optimize the activity for figuring out this output, rather than the whole strategy, of which right now you only need this aspect and no more.
Edit: Though, you don't need "input" to hold the rest of the strategy.
I was having trouble understanding what strategy couldn't be captured by a function X -> Y. After all, what could possibly determine the output of an algorithm other than its source code and whatever input it remembers getting on that particular run? Just to be clear, do you now agree that every strategy is captured by some function f: X -> Y mapping inputs to outputs?
One potential problem is that there are infinitely many input-output mappings. The agent can't assume a bound on the memory it will have, so it can't assume a bound on the lengths of inputs X that it will someday need to plug into an input-output mapping f.
Unlike the case where there are potentially infinitely many programs P1, P2, . . ., it's not clear to me that it's enough to wrap up an infinte set I of input-output mappings into some finite program that generates them. This is because the UDT1.1 agent needs to compute a sum for every element of I. So, if the set I is infinite, the number of sums to be computed will be infinite. Having a finite description of I won't help here, at least not with a brute-force UDT1.1 algorithm.
Any infinite thing in any given problem statement is already presented to you with a finite description. All you have to do is transform that finite description of an infinite object so as to get a finite description of a solution of your problem posed about the infinite object.