Alexandros comments on A cynical explanation for why rationalists worry about FAI - Less Wrong

25 Post author: aaronsw 04 August 2012 12:27PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (179)

You are viewing a single comment's thread. Show more comments above.

Comment author: Alexandros 05 August 2012 05:21:51AM 0 points [-]

So if I read correctly, someone at SI (Eliezer, even) had an original insight into cutting-edge AGI research, one strong enough to be accepted by other cutting-edge AGI researchers, and instead of publishing a proof of it, which was trivial, simply gave it away and some students finally proved it? Or were the discoveries independent?

Because if it the first, SI let a huge, track-record-building accomplishment slip through its hands. A paper like that alone would do a lot to answer Holden's criticism.

Comment author: CarlShulman 05 August 2012 05:31:04AM *  5 points [-]

Or were the discoveries independent?

I'm not sure. If they were connected, it was probably by way of the grapevine via the Schmidhuber/Hutter labs.

SI let a huge, track-record-building accomplishment slip through its hands

Meh, people wouldn't have called it huge, and it isn't, particularly. It would have signaled some positive things, but not much.

Comment author: timtyler 05 August 2012 01:21:33PM 4 points [-]

So if I read correctly, someone at SI (Eliezer, even) had an original insight into cutting-edge AGI research, one strong enough to be accepted by other cutting-edge AGI researchers, and instead of publishing a proof of it, which was trivial, simply gave it away and some students finally proved it?

Surely Hutter was aware of this issue back in 2003:

Another problem connected, but possibly not limited to embodied agents, especially if they are rewarded by humans, is the following: Sufficiently intelligent agents may increase their rewards by psychologically manipulating their human “teachers”, or by threatening them. This is a general sociological problem which successful AI will cause, which has nothing specifically to do with AIXI. Every intelligence superior to humans is capable of manipulating the latter.