= 27d567bc2d48d9f40e9ccc20ea370b15)
XiXiDu comments on What I would like the SIAI to publish - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (218)
The question is not what privileges doing what it is told but why it would do what it is not told? A crude mechanical machine has almost no freedom, often it can only follow one pathway. An intelligent machine on the other hand has much freedom, it can follow infinitely many pathways. With freedom comes choice and the necessity to decide, to follow one pathway but not others. Here you assume that a general intelligence will follow a pathway of self-improvement. But I do not think that intelligence implies self-improvement and further that following a pathway that leads an intelligence to optimize will be taken without it being a explicit specified goal. And that is where I conclude that from a certain number of AGI projects not all will follow the pathway of unbounded, dangerous self-improvement as there are more pathways to follow which lead any given general intelligence to be impassive or hold.
If you've read the thread above you'll see that my incentive is not to propose that there is no serious risk but that it is not inevitable that any AGI will turn out to be an existential risk. I want to propose that working on AGI carefully can help us better understand and define friendliness. I propose that the risk to carefull work on AGI is justified and does not imply our demise in any case.
Because planning consists in figuring out instrumental steps on your own.
If we are talking about a full-fledged general intelligence here (Skynet), there's no arguing against any risk. I believe all we disagree about are definitions. That there are risks from advanced real-world (fictional) nanotechnology is indisputable. I'm merely saying that what researchers are working on is nanotechnology with the potential to lead to grey goo scenarios but that there is no inherent risk that any work on it will lead down the same pathway.
It is incredible hard to come up with an intelligence that knows what planning conists of and to know and care to be able to judge what step is instrumental. This won't just happen accidently and will likely necessitate knowledge sufficient to be able to set scope boundaries as well. Again, this is not an argument that there is no risk but that it is not as strong as some people believe it to be.
Please keep focus, which is one of the most important tools. The above paragraph is unrelated to what I addressed in this conversation.
Review the above paragraph: what you are saying is that AIs are hard to build. But of course chess AIs do plan, to give an example. They don't perform only the moves they are "told" to perform.
What I am talking about is that full-fledged AGI is incredible hard to achieve and that therefore most of all AGI projects will fail on something other than limiting the AGI's scope. Therefore it is not likely that work on AGI is as dangerous as proposed.
That is, it is much more likely that any given chess AI will fail to beat a human player than that it will win. Still the researchers are working on chess AI's and the chess AI's will suit the definition of a general chess AI. Yet to get everything about a chess AI exactly right to beat any human but fail to implement certain performance boundaries (e.g. strength of its play or that it will overheat its CPU's etc.) is an unlikely outcome. It is more likely that it will be good at chess but not superhuman, that it will fail to improve, slow or biased than that it will succeed on all of the previous and additionally leave its scope boundaries.
So the discussion is about if the idea that any work on AGI is incredible dangerous is strong or if it can be weakened.