MrMind comments on Tiling Agents for Self-Modifying AI (OPFAI #2) - Less Wrong

55 Post author: Eliezer_Yudkowsky 06 June 2013 08:24PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (260)
Tags: math godel lob opfai

Comments (260)

You are viewing a single comment's thread. Show more comments above.

Comment author: MrMind 07 June 2013 07:07:53AM 0 points [-]

Page 4, footnote 8: I don't think it's true that only stronger systems can prove weaker systems consistent. It can happen that system A can prove system B consistent and A and B are incomparable, with neither stronger than the other.

That is strictly correct, but not relevant for self-improving AI. You don't want father AI that cannot prove everything that the child AI can prove. Maybe the footnote should be edited in this sense.

Comment author: jsteinhardt 07 June 2013 01:58:54PM *  0 points [-]

Well, if A can prove everything B can, except for con(A), and B can prove everything A can, except for con(B), then you're relatively happy.

ETA: retracted (thanks to Joshua Z for pointing out the error).

Comment author: JoshuaZ 08 June 2013 02:17:29AM 1 point [-]

I don't think this can't happen, since A has proven Con(B), then it can now reason using system B for consistency purposes and get from the fact that B proves Con(A) to get A proving Con(A), which is bad.

Comment author: jsteinhardt 08 June 2013 03:34:03AM 1 point [-]

Thanks for pointing this out. My mathematical logic is rusty.

Powered by Reddit Powered by Reddit