= 27d567bc2d48d9f40e9ccc20ea370b15)
gRR comments on Formulas of arithmetic that behave like decision agents - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (33)
Continuing from here...
Is there a particular reason to define U as a constant? What if it was defined as a function, taking the agent function as its argument? For example:
U(#X) =
(a, 0) if X(#U,0) = C;
(b, 0) if X(#U,0) = D
Then A could be defined as:
A(#X, i) = y, such that there exists #z, such that z(#X, i)=y and forall #z' pi(X(#z)) >= pi(X(#z'))
[Some uniquness condition on y can be added for the cases where several actions may have equal maximal utility]
Then it should be provable that a>b => A(#U,0)=C and a<b => A(#U,0)=D
This should also work with the toy PD. (But not with the real one - the agent's definition won't work because it is impossible to impersonate some other agent if you have to present your own Goedelian number).
The problem with your definition of U(#X) is that it is not a recursive function. There is no recursive function that takes the Gödel number of a function and computes the value of that function. So there is no formula representing U, and so U doesn't have a Gödel number.
Huh? Of course, there is - the Universal Turing Machine. You were probably thinking of the halting problem, but here we can just assume every function halts.
[The agent idea was naive though. I was young and stupid when I wrote the grandfather comment :)]
The agent described in the OP is definitely not a Turing machine that halts.
Hmm. The agent is a recursive function, so it is equivalent to a Turing machine that halts, no?
But I see your point. There is no simple formula, so if nothing else, it's inelegant.
Whoops! I said in the OP that the precursor to A (before diagonalization) is a recursive function, but it's not. It's not computable because it has to check whether several propositions are theorems of Peano arithmetic, and also because of that existential quantifier in the definition of A. If A itself were recursive, then it would provably (in PA) cooperate or defect; but it cooperates or defects unprovably (in PA). I'll edit the OP to make it correct.
The rest of the post still works because all the functions are representable — they can be represented by formulas in arithmetic — even if they're not recursive. Thanks for bringing this to my attention.
EDIT: Oy, they're not even "representable" in the technical sense. They're only definable.