I had a waking nightmare: I know some of you reading this just went "Oh great, here we go..." but bear with me. I am a man who loves to create and build, it is what I have dedicated my life to. One day because of the Less Wrong community I was prompted to ask "What if they are successful in creating an artificial general intelligence whose intellect dwarfs our own?"
My mind raced and imagined the creation of an artificial mind designed to be creative, subservient to man but also anticipate our needs and desires. In other words I imagined if current AGI engineers accomplished the creation of the greatest thing ever. Of course this machine would see how we loathe tiresome repetitive work and design and build for us a host of machines to do it for us. However then the horror at the implication of this all set in. The AGI will become smarter and smarter through its own engineering and soon it will anticipate human needs and produce things no human being could dream of. Suddenly man has no work to do, there is no back breaking labor to be done nor even the creative glorious work of engineering, exploring and experimentation. Instead our army of AGI has robbed us of that.
At this moment I certainly must express that this is not a statement amounting to "Lets not make AGI" for we all know AGI are coming. Then what is my point in expressing this? To express a train of thought that results in questions that have yet to be answered in the hopes that in depth discussion may shed some light.
I realized that the only meaning for man in a world run by AGI would actually be to order the AGI to make man himself better. Instead of focusing on having the AGI design a world for us, use that intellect that we could not before modification compare with to design a means to put us on its own level. In other words, the goal of creating an AGI should not to be to create an AGI but to make a tool so powerful we can use it to command man to be better. Now, I'm quite certain the audience present here is well aware of transhumanism. However, there are some important questions to be answered on the subject:
Mechanical or Biological modification? I know many would think "Are you stupid?! Of course cybernetics would be better than genetic alteration!" Yet the balance of advantages is not as clear as one would think. Lets consider cybernetics for a moment: Many would require maintenance, they would need to be designed and manufactured and therefore quite expensive. They also would need to be installed. Initially, possibly for decades only the rich could afford such a thing creating a titanic rift in power. This power gap of course will widen the already substantial resentment between the regular folk and the rich thereby creating political and social uncertainty which we can ill afford in a world with the kind of destructive power nuclear arms present.
Genetic alteration comes with a whole new set of problems. A titanic realm of genetic variables in which tweaking one thing may unexpectedly alter and damage another thing. Research in this area could potentially take much longer due to experimentation requirements. However the advantage is that genetic alteration can be accomplished with the help of virus in controlled environments. There would be no mechanic required to maintain the new being we have created and if designed properly the modifications can be passed down to the next generation. So instead of having to pay to upgrade each successive generation we instead only have to pay to upgrade one single generation. The rich obviously would still be the first ones to afford this procedure, however it could quickly spread across the globe due its potentially lower cost nature once development costs have been seen to. However, the problem is that we would be fundamentally and possibly irreversibly be altering our genetic code. Its possible to keep a gene bank so we have a memory of what we were in the hopes we could undo the changes and revert if the worst happened yet that is not the greatest problem with this path. We cannot even get the public to accept the concept of genetically altered crops, how can we get a world to accept its genes being altered? The sort of instability created by trying to push such a thing too hard, or the power gap created by those who have upgraded and who have not can again cause substantial instability that is globally dangerous.
So now I ask you, the audience. Genetic or cybernetic? How would we solve the political problems associated with both? What are the problems with both?
I don't disagree with you- this would, indeed, be a sad fate for humanity, and certainly a failed utopia. But the failing here is not inherent to the idea of an AGI that takes action on its own to improve humanity- it's of one that doesn't do what we actually want it to do, a failure to actually achieve friendliness.
Speaking of what we actually want, I want something more like what's hinted at in the fun theory sequence than one that only slowly improves humanity over decades, which seems to be what you're talking about here. (Tell me if I misunderstood, of course.)
You actually hit the nail on the head in terms of understanding the AGI I was referencing.
I thought about problems such as why would a firm researching crop engineering to solve world hunger bother with paying a full and very expensive staff? Wouldn't an AGI that not only crunches the numbers but manages mobile platforms for physical experimentation be more cost effective? The AGI would be smarter and run around the clock testing, postulating and experimenting. Researchers would quickly find themselves out of a job if the ideal AGI were born for this purp... (read more)