RSI capabilities could be charted, and are likely to be AI-complete.
What does RSI stand for?
Lately I've been listening to audiobooks (at 2x speed) in my down time, especially ones that seem likely to have passages relevant to the question of how well policy-makers will deal with AGI, basically continuing this project but only doing the "collection" stage, not the "analysis" stage.
I'll post quotes from the audiobooks I listen to as replies to this comment.
More (#3) from Better Angels of Our Nature:
...let’s have a look at political discourse, which most people believe has been getting dumb and dumber. There’s no such thing as the IQ of a speech, but Tetlock and other political psychologists have identified a variable called integrative complexity that captures a sense of intellectual balance, nuance, and sophistication. A passage that is low in integrative complexity stakes out an opinion and relentlessly hammers it home, without nuance or qualification. Its minimal complexity can be quantified by counting words like absolutely, always, certainly, definitively, entirely, forever, indisputable, irrefutable, undoubtedly, and unquestionably. A passage gets credit for some degree of integrative complexity if it shows a touch of subtlety with words like usually, almost, but, however, and maybe. It is rated higher if it acknowledges two points of view, higher still if it discusses connections, tradeoffs, or compromises between them, and highest of all if it explains these relationships by reference to a higher principle or system. The integrative complexity of a passage is not the same as the intelligence of the person who wrote it, but the
Okay. In this comment I'll keep an updated list of audiobooks I've heard since Sept. 2013, for those who are interested. All audiobooks are available via iTunes/Audible unless otherwise noted.
Outstanding:
Worthwhile if you care about the subject matter:
A process for turning ebooks into audiobooks for personal use, at least on Mac:
Personal and tribal selfishness align with AI risk-reduction in a way they may not align on climate change.
This seems obviously false. Local expenditures - of money, pride, possibility of not being the first to publish, etc. - are still local, global penalties are still global. Incentives are misaligned in exactly the same way as for climate change.
RSI capabilities could be charted, and are likely to be AI-complete.
This is to be taken as an arguendo, not as the author's opinion, right? See IEM on the minimal conditions for takeoff. Albeit if &q...
(I don't have answers to your specific questions, but here are some thoughts about the general problem.)
I agree with most of you said. I also assign significant probability mass to most parts of the argument for hope (but haven't thought about this enough to put numbers on this), though I too am not comforted on these parts because I also assign non-small chance to them going wrong. E.g., I have hope for "if AI is visible [and, I add, AI risk is understood] then authorities/elites will be taking safety measures".
That said, there are some steps in...
I personally am optimistic about the world's elites navigating AI risk as well as possible subject to inherent human limitations that I would expect everybody to have, and the inherent risk. Some points:
I've been surprised by people's ability to avert bad outcomes. Only two nuclear weapons have been used since nuclear weapons were developed, despite the fact that there are 10,000+ nuclear weapons around the world. Political leaders are assassinated very infrequently relative to how often one might expect a priori.
AI risk is a Global Catastrophic Risk i
The argument from hope or towards hope or anything but despair and grit is misplaced when dealing with risks of this magnitude.
Don't trust God (or semi-competent world leaders) to make everything magically turn out all right. The temptation to do so is either a rationalization of wanting to do nothing, or based on a profoundly miscalibrated optimism for how the world works.
/doom
I think there's a >15% chance AI will not be preceded by visible signals.
Aren't we seeing "visible signals" already? Machines are better than humans at lots of intelligence-related tasks today.
Which historical events are analogous to AI risk in some important ways? Possibilities include: nuclear weapons, climate change, recombinant DNA, nanotechnology, chloroflourocarbons, asteroids, cyberterrorism, Spanish flu, the 2008 financial crisis, and large wars.
Cryptography and cryptanalysis are obvious precursors of supposedly-dangerous tech within IT.
Looking at their story, we can plausibly expect governments to attempt to delay the development of "weaponizable" technology by others.
These days, cryptography facilitates international trade. It seems like a mostly-positive force overall.
One question is whether AI is like CFCs, or like CO2, or like hacking.
With CFCs, the solution was simple: ban CFCs. The cost was relatively low, and the benefit relatively high.
With CO2, the solution is equally simple: cap and trade. It's just not politically palatable, because the problem is slower-moving, and the cost would be much, much greater (perhaps great enough to really mess up the world economy). So, we're left with the second-best solution: do nothing. People will die, but the economy will keep growing, which might balance that out, because ...
Here are my reasons for pessimism:
There are likely to be effective methods of controlling AIs that are of subhuman or even roughly human-level intelligence which do not scale up to superhuman intelligence. These include for example reinforcement by reward/punishment, mutually beneficial trading, legal institutions. Controlling superhuman intelligence will likely require qualitatively different methods, such as having the superintelligence share our values. Unfortunately the existence of effective but unscalable methods of AI control will probably lull el
Congress' non-responsiveness to risks to critical infrastructure from geomagnetic storms, despite scientific consensus on the issue, is also worrying.
Even if one organization navigates the creation of friendly AI successfully, won't we still have to worry about preventing anyone from ever creating an unsafe AI?
Unlike nuclear weapons, a single AI might have world ending consequences, and an AI requires no special resources. Theoretically a seed AI could be uploaded to Pirate Bay, from where anyone could download and compile it.
The use of early AIs to solve AI safety problems creates an attractor for "safe, powerful AI."
What kind of "AI safety problems" are we talking about here? If they are like the "FAI Open Problems" that Eliezer has been posting, they would require philosophers of the highest (perhaps even super-human) caliber to solve. How could "early AIs" be of much help?
If "AI safety problems" here do not refer to FAI problems, then how do those problems get solved, according to this argument?
@Lukeprog, can you
(1) update us on your working answers the posed questions in brief? (2) your current confidence (and if you would like to, by proxy, MIRI's as an organisation's confidence in each of the 3:
Elites often fail to take effective action despite plenty of warning.
I think there's a >10% chance AI will not be preceded by visible signals.
I think the elites' safety measures will likely be insufficient.
Thank you for your diligence.
There's another reason for hope in this above global warming: The idea of a dangerous AI is already common in the public eye as "things we need to be careful about." A big problem the global warming movement had, and is still having, is convincing the public that it's a threat in the first place.
Who do you mean by "elites". Keep in mind that major disruptive technical progress of the type likely to precede the creation of a full AGI tends to cause the type of social change that shakes up the social hierarchy.
Combining the beginning and the end of your questions reveals an answer.
Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of [nuclear weapons, climate change, recombinant DNA, nanotechnology, chloroflourocarbons, asteroids, cyberterrorism, Spanish flu, the 2008 financial crisis, and large wars] just fine?
Answer how just fine any of these are any you have analogous answers.
You might also clarify whether you are interested in what is just fine for everyone, or just fine for the elites, or just fine for the AI in question. The answer will change accordingly.
More (#2) from Dirty Wars:
According to the ICRC, some of the prisoners were bounced around to different black sites for more than three years, where they were kept in “continuous solitary confinement and incommunicado detention. They had no knowledge of where they were being held, no contact with persons other than their interrogators or guards.” The US personnel guarding them wore masks. None of the prisoners was ever permitted a phone call or to write to inform their families they had been taken. They simply vanished.
During the course of their imprisonment, some of the prisoners were confined in boxes and subjected to prolonged nudity—sometimes lasting for several months. Some of them were kept for days at a time, naked, in “stress standing positions,” with their “arms extended and chained above the head.” During this torture, they were not allowed to use a toilet and “had to defecate and urinate over themselves.” Beatings and kickings were common, as was a practice of placing a collar around a prisoner’s neck and using it to slam him against walls or yank him down hallways. Loud music was used for sleep deprivation, as was temperature manipulation. If prisoners were perceived to be cooperating, they were given clothes to wear. If they were deemed uncooperative, they’d be stripped naked. Dietary manipulation was used—at times the prisoners were put on liquid-only diets for weeks at a time. Three of the prisoners told the ICRC they had been waterboarded. Some of them were moved to as many as ten different sites during their imprisonment. “I was told during this period that I was one of the first to receive these interrogation techniques, so no rules applied,” one prisoner, taken early on in the war on terror, told the ICRC. “I felt like they were experimenting and trying out techniques to be used later on other people.”
And:
While TF-121 was given a mission to kill or capture Osama bin Laden and Saddam Hussein by the spring of 2004, Washington was increasingly focused on Iraq. Veteran intelligence officials identify this period as a turning point in the hunt for bin Laden. At a time when JSOC was asking for more resources and permissions to pursue targets inside of Pakistan and other countries, there was a tectonic shift toward making Iraq the numberone priority.
The heavy costs of that strategic redirection to the larger counterterrorism mission were of deep concern to Lieutenant Colonel Anthony Shaffer, a senior military intelligence officer who was CIA trained and had worked for the DIA and JSOC. Shaffer ran a task force, Stratus Ivy, that was part of a program started in the late 1990s code-named Able Danger. Utilizing what was then cutting-edge “data mining” technology, the program was operated by military intelligence and the Special Operations Command and aimed at identifying al Qaeda cells globally. Shaffer and some of his Able Danger colleagues claimed that they had uncovered several of the 9/11 hijackers a year before the attacks but that no action was taken against them. He told the 9/11 Commission he felt frustrated when the program was shut down and believed it was one of the few effective tools the United States had in the fight against al Qaeda pre-9/11. After the attacks, Shaffer volunteered for active duty and became the commander of the DIA’s Operating Base Alpha, which Shaffer said “conducted clandestine antiterrorist operations” in Africa. Shaffer was running the secret program, targeting al Qaeda figures who might flee Afghanistan and seek shelter in Somalia, Liberia and other African nations. It “was the first DIA covert action of the post–Cold War era, where my officers used an African national military proxy to hunt down and kill al Qaeda terrorists,” Shaffer recalled.
Like many other experienced intelligence officers who had been tracking al Qaeda prior to 9/11, Shaffer believed that the focus was finally placed correctly on destroying the terror network and killing or capturing its leaders. But then all resources were repurposed for the Iraq invasion. “I saw the Bush administration lunacy up close and personal,” Shaffer said. After a year and a half of running the African ops, “I was forced to shut down Operating Base Alpha so that its resources could be used for the Iraq invasion.”
Shaffer was reassigned as an intelligence planner on the DIA team that helped feed information on possible Iraqi WMD sites to the advance JSOC teams that covertly entered Iraq ahead of the invasion. “It yielded nothing,” he alleged. “As we now know, no WMD were ever found.” He believed that shifting the focus and resources to Iraq was a grave error that allowed bin Laden to continue operating for nearly another decade. Shaffer was eventually sent to Afghanistan, where he would clash with US military leaders over his proposals to run operations into Pakistan to target the al Qaeda leaders who were hiding there.
And:
The task force’s operations, Exum said, were “very compartmentalized, very stove-piped.” JSOC was creating a system where its intelligence operations were feeding its action and often that intelligence would not be vetted by anyone outside of the JSOC structure. The priority was to keep hitting targets. “The most serious thing is the abuse of power that that allows you to do,” said Wilkerson, the former chief of staff to Powell. He continued: "You go in and you get some intelligence, and usually your intelligence comes through this apparatus too, and so you say, ‘Oh, this is really good actionable intelligence. Here’s Operation Blue Thunder. Go do it.’ And they go do it, and they kill 27, 30, 40 people, whatever, and they capture seven or eight. Then you find out that the intelligence was bad and you killed a bunch of innocent people and you have a bunch of innocent people on your hands, so you stuff ’em in Guantánamo. No one ever knows anything about that. You don’t have to prove to anyone that you did right. You did it all in secret, so you just go to the next operation. You say, ‘Chalk that one up to experience,’ and you go to the next operation. And, believe me, that happened."
One open question in AI risk strategy is: Can we trust the world's elite decision-makers (hereafter "elites") to navigate the creation of human-level AI (and beyond) just fine, without the kinds of special efforts that e.g. Bostrom and Yudkowsky think are needed?
Some reasons for concern include:
But if you were trying to argue for hope, you might argue along these lines (presented for the sake of argument; I don't actually endorse this argument):
The basic structure of this 'argument for hope' is due to Carl Shulman, though he doesn't necessarily endorse the details. (Also, it's just a rough argument, and as stated is not deductively valid.)
Personally, I am not very comforted by this argument because:
Obviously, there's a lot more for me to spell out here, and some of it may be unclear. The reason I'm posting these thoughts in such a rough state is so that MIRI can get some help on our research into this question.
In particular, I'd like to know: