Pfft comments on An angle of attack on Open Problem #1 - Less Wrong

30 Post author: Benja 18 August 2012 12:08PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (84)

Comments (84)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 20 August 2012 03:00:53AM *  8 points [-]

I played around a bit with quining solutions, and came up with the following, which solves this toy problem fairly well:

def AI_Q2(n)(p):
- Look at the first n^^^n theorems of PA(n).
- if (one of them says "for each q, there exists an m such that p(q) != 'self-destruct' and p(q) = 'double down' implies AI_Q2(m)(q) = 'double down'"):
-- double down
- else:
-- take winnings

AI_Q2(3) should double down on AI_Q2(4) as well as AI_Q2(4^^^^4). (As well as variants that are provably equivalent to itself like speed/size-optimized versions of itself.) I sent this to Benja last night and he responded with (in part) "You've succeeded in making me uncertain whether quining approaches could actually be directly useful in solving the real problem (though it still doesn't seem likely)."

I agree it doesn't seem likely the real problem can be solved with quining approaches, but I'm post this solution here in case anyone can extend the idea further. At the very least it should be interesting to understand why quining approaches don't work on the real problem. What relevant aspect of the real problem isn't being captured by this toy problem?

Comment author: Pfft 22 August 2012 07:56:28PM *  3 points [-]

Do you gain anything by parameterizing on PA(n)? It seems like you could get a sound algorithm by replacing PA(n) in your definition by T, for any trustworthy system of axioms T. My first instinct was to pick T=ZFC, while you pick a dynamically changing subset of PA(omega), i.e. PA + a countably infinite sequence of consistency assertions.

Certainly PA(omega) is trustworthy, but there is nothing particularly canonical about it (you could equally well pick PA(omega)+CON(PA(omega))), and I don't think there is any loss of generality in picking the axioms up-front.

Comment author: Wei_Dai 22 August 2012 09:08:29PM 2 points [-]

I think you're right. Parameterizing on PA(n) doesn't really seem to do anything except perhaps give an illusion of self-improvement when there's really a hard limit of PA(omega) that the AI can't go beyond. I guess I choose it mostly because the OP was doing something similar.

Comment author: cousin_it 23 August 2012 03:45:31PM *  0 points [-]

If you just have PA(omega) and, say, parameterize on the proof limit instead, will your program double down on versions of itself that use weaker formal theories, not mentioned in your program?

Comment author: Pfft 23 August 2012 06:15:38PM *  0 points [-]

Yes. If you call it with the argument AIQ2_T (i.e. the same algorithm but with T instead of PA(omega)), it will accept if it can find a proof that forall q,

AIQ2_T(q) != 'self-destruct'

(which is easy) and

AIQ2_T(q) == 'double' ==> AIQ2(q)=='double'

The latter condition expands to

PA(omega) |- someFormula ==> T |- someFormula

which may be quite easy to show, depending on T.

Basically, the Quining approach avoids trouble from the 2nd incompleteness theorem by using relative consistency instead of consistency.

Comment author: cousin_it 23 August 2012 09:22:15PM *  1 point [-]

It seems to me that the two programs don't have the same "someFormula", because each program's formula uses a quined description of that program, but not the other one. Or maybe I'm wrong. Can you expand what you mean by "someFormula"?

Comment author: Pfft 23 August 2012 10:37:53PM 0 points [-]

Oops.

Powered by Reddit Powered by Reddit