The formalism presented in this post turned out to be erroneous (as opposed to the formalism in the previous post). The problem is that the step in the proof of the main proposition in which the soundness schema is applied cannot be generalized to the ordinal setting since we don't know whether ακ is a successor ordinal so we can't replace it by ακ'=ακ-1. I'm not deleting this post primarily to preserve the useful discussion in the comments.
Followup to: Parametric polymorphism in updateless intelligence metric
In the previous post, I formulated a variant of Benja's parametric polymorphism suitable for constructing updateless intelligence metrics. More generally, this variants admits agents which are utility maximizers (in the informal sense of trying their best to maximize a utility function, not in the formal sense of finding the absolutely optimal solution; for example they might be "meliorizers" to use the terminology of Yudkowsky and Herreshoff) rather than satisficers. The agents using this formalism labor under a certain "delusion", namely, since they believe that κ ("the number of ineffable mystery") is an actual finite number (whereas it is secretly infinite, in a sense), they think that the chain of tiling agents is effectively finite as well (since at some point agent #n in the chain will discover that κ > n is false and will fail to construct a subsequent agent of the same "epistemic strength"). The same problem exists in Weaver's intuitionistic assertability predicate formalism.
To overcome this limitation, I suggest to extend κ's semantics from natural numbers to elements of a certain recursive ordinal (at least morally; technically it is done a bit differently, see below). In Benja's original formulation this doesn't appear a valid option, since κ is interpreted as a time interval. However, in my variant κ is just an abstract parameter queries about which can be directed to a special "oracle", so there is no such limitation.
Formalism
Fix α a recursive ordinal and αi a system of notations for ordinals smaller than α such that α0=0, the function f defined by αf(i)=αi+1 is recursive and the bit-valued function g(i,j) which equals 1 iff αi < αj is also recursive. We also assume that
(0) T |- For all i: g(i,f(i))=1
where T is our starting theory. We define Tκ(α) to be T extended by the symbol κ (which stands for the index of the ordinal, not the ordinal itself) and the following self-referential soundness schema:
(1) For all formulae φ: For all n, i: ((@_Tκ(α) "φ(n)" and g(i,κ)=1) -> φ(n)[κ\i])
Tκ(α) is proven to be sound analogously to Yudkowsky and Herreshoff, with transfinite induction over ordinals below α replacing the ordinary induction.
The licensing condition for agent A0 to perform action b0 is
(2) Exists n: Tκ(α) |- (g(n,κ)=1 -> (A0()=b0 -> G))
The property which is sufficient for agent B1 to be creatable by agent A0 is
(3) Tκ(α) |- For all b1: (B1()=b1 -> (b1=0 or Exists n: (g(n,κ)=1 and @_Tκ(α) "(g(n,f(κ))=1 and B1()[κ\f(κ)]=b1) -> G[κ\f(κ)]")))
The proof that (3) indeed enables tiling is analogous to the previous post (but requires using (0)).
Discussion
In order for A0 to know that the tiling chain is infinite, it is enough for her to verify ακ >= ω. In order for her to know that A1 is going to know the tiling chain is infinite, she needs to verify ακ >= ω+1. In order for her to know that all agents are going to know the tiling chain is infinite, she needs to verify ακ >= 2ω. In order for her to know that all agents are going to know that, she needs to verify ακ >= 3ω et cetera.
It remains to decide which ordinal should we actually use. My intuition is that the correct ordinal is the least α with the property that α is the proof-theoretic ordinal of Tκ(α) extended by the axiom schema {g(i,κ)=1}. This seems right since the agent shouldn't get much from ακ > β for β above the proof theoretic ordinal. However, a more formal justification is probably in order.
Hi So8res, thx a lot for reading the post and commenting on it!
I understand the difference. I just think the terminology is a bit confusing, since many people use "utility maximizer" in the informal sense of "a gadget that tries to get as much utility as it can" rather than "something that finds the absolutely optimal strategy". I think that ultimately the solution is going to be "compute expected utility for a small number of strategies and choose the best". One of the strategies will be "think more" (since the expectation value contains logical uncertainty which by definition depends on the amount of computing resources invested into evaluating it). Now, since the actual space of possible actions is large, we need to break it down in tree-like fashion so that the choice from a large set is replaced by many choices from small sets. But all of that is very tangential to the current post.
You're describing Fallenstein's original PP which I understand. My variant works differently. It allows the agent to prove kappa > n for any n. In this respect, it is similar to Weaver's formalism.
The quantifier is in the right place. You cannot choose n large enough since T_kappa(alpha) cannot prove a_n >= a_k for any particular n. It can only prove "Exists n: a_n >= a_k" which is why the quantifier has to be outside the turnstile. The physical meaning of this licensing condition is as follows. The agent is a theorem proving machine that reasons using T_kappa(alpha). In addition, it has access to a special "oracle" which is a black-box from the agent's POV and can answer questions of the form "a_n < a_k?". The oracle secretly always answers affirmatively (although the agents is sure of the contrary). Thus the agent can use the oracle's answers and the n outside the turnstile corresponds to the largest a_n about which the oracle was asked.
Unfortunately I got too lazy for Latex. In part it was an irrational response to getting little feedback to my previous post. Probably I have to fix it.
No, since there is no decreasing chain of ordinals here. Instead, we got a "twist" inserted in the oracle. After m steps it answers the question "a_n + m < a_k?" instead of answering "a_n < a_k" like the (new) agent thinks.
I'll be very glad to answer any further questions that you have!
I have to admit to my shame that I haven't read that paper. I will surely do it. Note though that my motivation was adapting the framework to work for utility expectation values instead of satificing and in particular for my updateless intelligence formalism.
Well, I mostly hoped the posts on LW will draw some attention. However, I will be very grateful for any advice how to get people in MIRI interested. Thx again!
This is a common sentiment, but note that a meliorizer is far from a maximizer even in the non-optimal sense. A meliorizer doesn't do any intelligent navigation of the search space, it just takes the first thing it can find which is better than its fallback plan.
... (read more)