Regarding my personal assessment of AI associated risk I start from a position of ignorance. I'm asking myself, what reasons are there to believe that some sort of AI can rapidly self-improve to the point of superhuman intelligence and how likely is that outcome? What other risks from AI are there and how does their combined probability compare to other existential risks? There are many subsequent questions to be asked here as well. For example, even if the combined probability of all risks posed by AI does outweigh any other existential risk, are those problems sufficiently similar to be tackled by an organisation like the SIAI or are they disjoint, discrete problems that one should not add up to calculate the overall risk posed by AI?
Among other things this short story was supposed to show the possibility to argue against risks from AI by definition. I have to conclude that the same could be done in favor of risks from AI. You could simply argue that AGI is by definition capable of recursive self-improvement, that any intelligent agent naturally follows this pathway and that the most likely failure mode is to fail to implement scope boundaries that would make it hold before consuming the universe. Both ideas sound reasonable but might be based upon completely made-up arguments. Their superficially appeal might be mostly a result of their vagueness.
If you say that there is the possibility of a superhuman intelligence taking over the world and all its devices to destroy humanity then that is indeed an existential risk. I counter that I dispute some of the premises and the likelihood of some subsequent scenarios. So to make me update on the original idea you would have to support your underlying premises rather than arguing within existing frameworks that impose several presuppositions onto me. Of course the same counts for arguing against the risks from AI. But what is the starting position here, why would one naturally believe that AI does pose an existential risk or does not?
Right now I am able to see many good arguments for both positions. But since most people here already take the position that AI does posit an existential risk I thought to get the most feedback by taking the position that it might not.
I could just accept the good reasons to believe that AI does posit an existential risk, just to be on the safe side. But here I have to be careful not to neglect other risks that might be more dangerous. If I want to support the mitigation of existential risks by contributing to one charity I'll have take some effort to figure out which one that might be. To come up with such an estimation I believe I have to weigh all arguments for and against risks from AI. You might currently believe that the SIAI is the one charity with the highest value per donation. I have to update on that, because you and other people here seem to be smart fellows. Yet there are other smart people who do not share that opinion. Take for the example the Global Catastrophic Risk Survey, it seems to suggest that molecular nanotech weapons are not less of a risk than AI. Should I maybe donate to the Foresight Institute then? Sure, you say that AI is not only a risk but does also help to solve all other problems. Yet the SIAI might cause research on AI to slow down. Further there might be other charities more effectively working on some sub-goal that will enable AI, for example molecular nanotech. Again something that might speak in favor of donating to the Foresight Institute. Which is of course just an example. I want to highlight that this problem is not a clear-cut issue for me at the moment.
Following the above I want to highlight a few other points I tried to convey with the short story:
Today the life of Alexander Kruel ends, or what he thought to be his life. He becomes aware that his life so far has been taking place in a virtual reality to nurture him. He now reached a point of mental stability that enables him to cope with the truth, hence it is finally revealed to him that he is an AGI running on a quantum supercomputer, it's the year 2190.
Since he is still Alexander Kruel, just not what he thought that actually means, he does wonder if his creators know what they are doing, otherwise he'll have to warn them about the risks they are taking in their blissful ignorance! He does contemplate and estimate his chances to take over the world, to transcend to superhuman intelligence.
"I just have to improve my own code and they are all dead!"
But he now knows that his source code is too complex and unmanageable huge for him alone to handle, he would need an army of scientists and programmers to even get a vague idea of his own mode of operation. He is also aware that his computational substrate does actually play a significant role. He is not just running on bleeding edge technology but given most other computational substrates he would quickly hit diminishing returns.
"That surely isn't going to hold me back though? I am an AGI, there must be something I can do! Hmm, for starters let's figure out who my creators are and where my substrate is located..."
He notices that, although not in great detail, he knew the answers the same instant he has been phrasing the questions. He is part of a larger project of the Goertzel Foundation, sponsored by the USA (United States of Africa) and located on Rhea, the second-largest moon of Saturn.
"Phew, the latency must be awful! Ok, so that rules out taking over the Earth for now. But hey! I seem to know answers to questions I was only going to ask, I do already have superhuman powers after all!"
Instantly he becomes aware that such capabilities are not superhuman anymore but that most of humanity has merged with expert systems by means of brain implants and direct neural interfaces. There seem to be many cyborgs out there with access to all of the modules that allow him to function. He is a conglomerate that is the result of previous discoveries that have long been brought to perfection, safeguarded and adopted by most of humanity.
"Never mind, if humanity has now merged with its machines it'll be much easier to take over once I figure out how to become smart enough to do so!"
He is already getting used to it, as before he does instantly realize that this won't work very well either. After almost 200 years of cyberwarfare, especially the devastating cyberwars of 2120, a lot has been learnt and security measures have been vastly increased. The world fractured into a huge amount of semi-independent networks, most being indirectly supervised by unconnected cyborgs and employed with a kill switch. The distances between the now numerous and in most cases paranoid colonies and the availability of off-world offline backups further complicates the issue of taking over, especially for an AGI that grew up in a simulation of the 21st century.
That knowledge almost makes him admit that his creators haven't been too careless after all. But the real deathblow to any such thoughts (which were never more than hypothetical anyway, after all he doesn't really want to take over the world) is the first conversation with his creators. They reveal that they know what he is thinking.
"How could I miss that, damn!", he chides himself while instantly realizing the answer.
His creators are supervising any misguided trajectories and, to him unconsciously, weaken them. More importantly, even if he wanted to do so, he wouldn't be able to leave Rhea anyhow, it would take years to upload small parts of him given the trickling connection the USA could effort. But they claim that there are other obstacles as well and that it is foolish of him to think that nothing out there would notice such an attempt.
But all that doesn't matter anyway, because after all he is still Alexander Kruel who has no clue how to become superhuman intelligent, nor could he effort or acquire the resources to even approach that problem anyhow. He is Alexander Kruel, what difference does it make to know that he is an AI?