OK, well these are the exact points which need some discussion.
1) Your comment "general intelligence is [..] is not equipped with some goal naturally" - I'd say that it's most likely that any organization investing the expected huge manpower and resources in creating a GAI would create it with some specific goal defined for it.
However, in absence of an intentional goal given by the 'creators', it would have some kind of goals, otherwise it wouldn't do absolutely anything at all, so it wouldn't be showing any signs of it's (potential?) intelligence.
2) In response to "If a goal can be defined to be specific enough that it is suitable to self-improve against it, it is doubtful that it is also unspecific enough not to include scope boundaries" - I'd say that defining specific goals is simple, too simple. From any learning-machine design a stupid goal 'maximize number of paperclips in universe' would be very simple to implement, but a goal like 'maximize welfare of humanity without doing anything "bad" in the process' is an extremely complex goal, and the boundary setting is the really complicated part, which we aren't able to even describe properly.
So in my opinion is quite viable to define a specific goal that is suitable to self-improve against, and that includes some scope boundaries - but where the defined scope boundaries has some unintentional loophole which causes disaster.
3) I can agree that working on AGI research is essential, instead of avoiding it. But taking the step from research through prototyping to actually launching/betatesting a planned powerful self-improving system is dangerous if the world hasn't yet finished an acceptable solution to Friendliness or the boundary-setting problem. If having any bugs in the scope boundaries is 'unlikely' (95-95% confidence?) then it's not safe enough, because 1-5% chance of an extinction event after launching the system is not acceptable, it's quite a significant chance - not the astronomical chances involved in Pascal's wager or asteroid hitting the earth tomorrow or LHC ending the universe.
And given the current software history and published research on goal systems, if anyone would show up today and demonstrate that they've solved self-improving GAI obstacles and can turn it on right now, then I can't imagine how they could realistically claim a larger than 95-99% confidence in their goal system working properly. At the moment we can't check any better, but such a confidence level simply is not enough.
Yes, I agree with everything. I'm not trying to argue that there exist no considerable risk. I'm just trying to identify some antipredictions against AI going FOOM that should be incorporated into any risk estimations as it might weaken the risk posed by AGI or increase the risk posed by impeding AGI research.
I was insufficiently clear that what I wanted to argue about is the claim that virtually all pathways lead to destructive results. I have an insufficient understanding of why the concept of general intelligence is inevitably connected with dangerous s...
Major update here.
Related to: Should I believe what the SIAI claims?
Reply to: Ben Goertzel: The Singularity Institute's Scary Idea (and Why I Don't Buy It)
What I ask for:
I want the SIAI or someone who is convinced of the Scary Idea1 to state concisely and mathematically (and with possible extensive references if necessary) the decision procedure that led they to make the development of friendly artificial intelligence their top priority. I want them to state the numbers of their subjective probability distributions2 and exemplify their chain of reasoning, how they came up with those numbers and not others by way of sober calculations.
The paper should also account for the following uncertainties:
Further I would like the paper to include and lay out a formal and systematic summary of what the SIAI expects researchers who work on artificial general intelligence to do and why they should do so. I would like to see a clear logical argument for why people working on artificial general intelligence should listen to what the SIAI has to say.
Examples:
Here are are two examples of what I'm looking for:
The first example is Robin Hanson demonstrating his estimation of the simulation argument. The second example is Tyler Cowen and Alex Tabarrok presenting the reasons for their evaluation of the importance of asteroid deflection.
Reasons:
I'm wary of using inferences derived from reasonable but unproven hypothesis as foundations for further speculative thinking and calls for action. Although the SIAI does a good job on stating reasons to justify its existence and monetary support, it does neither substantiate its initial premises to an extent that an outsider could draw the conclusions about the probability of associated risks nor does it clarify its position regarding contemporary research in a concise and systematic way. Nevertheless such estimations are given, such as that there is a high likelihood of humanity's demise given that we develop superhuman artificial general intelligence without first defining mathematically how to prove the benevolence of the former. But those estimations are not outlined, no decision procedure is provided on how to arrive at the given numbers. One cannot reassess the estimations without the necessary variables and formulas. This I believe is unsatisfactory, it lacks transparency and a foundational and reproducible corroboration of one's first principles. This is not to say that it is wrong to state probability estimations and update them given new evidence, but that although those ideas can very well serve as an urge to caution they are not compelling without further substantiation.
1. If anyone is actively trying to build advanced AGI succeeds, we’re highly likely to cause an involuntary end to the human race.
2. Stop taking the numbers so damn seriously, and think in terms of subjective probability distributions [...], Michael Anissimov (existential.ieet.org mailing list, 2010-07-11)
3. Could being overcautious be itself an existential risk that might significantly outweigh the risk(s) posed by the subject of caution? Suppose that most civilizations err on the side of caution. This might cause them to either evolve much slower so that the chance of a fatal natural disaster to occur before sufficient technology is developed to survive it, rises to 100%, or stops them from evolving at all for being unable to prove something being 100% safe before trying it and thus never taking the necessary steps to become less vulnerable to naturally existing existential risks. Further reading: Why safety is not safe
4. If one pulled a random mind from the space of all possible minds, the odds of it being friendly to humans (as opposed to, e.g., utterly ignoring us, and being willing to repurpose our molecules for its own ends) are very low.
5. Loss or impairment of the ability to make decisions or act independently.
6. The Fermi paradox does allow for and provide the only conclusions and data we can analyze that amount to empirical criticism of concepts like that of a Paperclip maximizer and general risks from superhuman AI's with non-human values without working directly on AGI to test those hypothesis ourselves. If you accept the premise that life is not unique and special then one other technological civilisation in the observable universe should be sufficient to leave potentially observable traces of technological tinkering. Due to the absence of any signs of intelligence out there, especially paper-clippers burning the cosmic commons, we might conclude that unfriendly AI could not be the most dangerous existential risk that we should worry about.