= 27d567bc2d48d9f40e9ccc20ea370b15)
MrMind comments on Could you tell me what's wrong with this? - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (64)
The problem of telling an AI to produce an FAI lies of course in the correct specification of the goal: - how do you specify such a goal without having already solved FAI? - How do you verify the correctedness of the solution if the goal is reached?
Plus, it's much harder then you seem to think to stop an AI from self-modifying: what if, for example, it hides secret instructions in the proposed solution for the next AI working on the problem? What if it hires a team of skilled individual to bust through the doors of the lab it's hosted and modify the code? These are just two solutions I came up within the first minute of thinking, and I'm not a genius nor surely an AI thinking about it for years on end.
Yeah, the first point started to make itself clear as I read through the comments. I think that the 'how do you verify...' part of your post is the most compelling. When we look at it, we don't know whether or not there will be any problems with this. But since it was produced by human level intellect, we should be able to understand it.
I had the idea of the second in my mind; I think that this scenario has a fair chance of happening, but a large group of humans would be able to stop the AIs attempts. Of course, they could still let it slip past their fingers, which would be Bad. I would say that that is preferable to having FAI turn out to be beyond us, and waiting until some tit decides to go ahead with an AGI anyway. But that does not make it a pleasant choice.
All in all, I think that this avenue wasn't so terrible, but it is still fairly risky. It seems that this is the sort of thing you'd use as maybe a Plan 3, or Plan 2 if someone came along and made it much more robust.