You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

cousin_it comments on SIAI Fundraising - Less Wrong Discussion

59 [deleted] 26 April 2011 08:35AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (116)

You are viewing a single comment's thread. Show more comments above.

Comment author: cousin_it 27 April 2011 05:08:14PM *  7 points [-]

Eliezer stated his reasons here:

...a constructive theory of the world's second most important math problem, reflective decision systems, is necessarily a constructive theory of seed AI; and constitutes, in itself, a weapon of math destruction, which can be used for destruction more quickly than to any good purpose. Any Singularity-value I attach to publicizing Friendly AI would go into explaining the problem. Solutions are far harder than this and will be specialized on particular constructive architectures.

So in a nutshell, he thinks solving decision theory will make building unfriendly AIs much easier. This doesn't sound right to me because we already have idealized models like Solomonoff induction or AIXI, and they don't help much with building real-world approximations to these ideals, so an idealized perfect solution to decision theory isn't likely to help much either. But maybe he has some insight that I don't.

Comment author: Wei_Dai 27 April 2011 06:22:20PM 3 points [-]

I think Eliezer must have changed his mind after writing those words, because his TDT book was written for public consumption all along. (He gave two reasons for not publishing it sooner: he wanted to see if a university would offer him a PhD based on it, and he was using DT as a problem to test potential FAI researchers.) I guess his current lack of participation in our DT mailing list is probably due to some combination of being busy with his books and lack of significant new insights.

Comment author: Nick_Tarleton 27 April 2011 06:36:20PM *  1 point [-]

I think TDT is different from the "reflective decision systems" he was talking about, which sounds like it refers to a theory specifically of self-modifying agents.

Comment author: wedrifid 27 April 2011 05:41:57PM 4 points [-]

a weapon of math destruction

That's the first time I noticed the pun. Good one. I want a tshirt.

Comment author: [deleted] 27 April 2011 05:21:38PM 1 point [-]

Ah. I see what he means, if you're talking about a) just the 'invariant under reflection' part and not Friendliness and b) you're talking about a strictly pragmatic tool. That makes sense.