have separate risk management systems which impose constraints and limits on what they can do.

If those risk management systems are themselves software, that doesn't really change the overall picture.

Yes, but that has nothing to do with AI:

If we're talking about "would companies place AI systems in a role where those systems could cost the company lots of money if they malfunctioned", then examples of AI systems having been placed in roles where they cost the company a lot of money have everything to do with the discussion.

When you say autonomous AIs, do you mean AIs that are autonomous and superinteligent?

AIs that are initially autonomous and non-superintelligent, then gradually develop towards superintelligence. (With the important caveat that it's unclear whether an AI needed to be generally superintelligent in order to pose a major risk for society. It's conceivable that superintelligence in some more narrow domain, like cybersecurity, would be enough - particularly in a sufficiently networked society.)

Do you think they could he deployed by basement hackers, or only by large organisations?

Hard to say. The way AI has developed so far, it looks like the capability might be restricted to large organizations with lots of hardware resources at first, but time will likely drive down the hardware requirements.

Do you think an organisation like the military or business has a motivation to deploy them?

Yes.

Do you agree that there are dangers to an FAI project that goes wrong?

Yes.

Do you have a plan B to cope with a FAI that goes rogue?

Such a plan would seem to require lots of additional information about both the specifics of the FAI plan, and also the state of the world at that time, so not really.

Do you think that having a AI potentially running the world is an attractive idea to a lot of people?

Depends on how we're defining "lots", but I think that the notion of a benevolent dictator has often been popular in many circles, who've also acknowledged its largest problems to be that 1) power tends to corrupt 2) even if you got a benevolent dictator, you also needed a way to ensure that all of their successors were benevolent. Both problems could be overcome with an AI, so on that basis at least I would expect lots of people to find it attractive. I'd also expect it to be considered more attractive in e.g. China, where people seem to be more skeptical towards democracy than they are in the West.

Additionally, if the AI wouldn't be the equivalent of a benevolent dictator, but rather had a more hands-off role that kept humans in power and only acted to e.g. prevent disease, violent crime, and accidents, then that could be attractive to a lot of people who preferred democracy.

Thank you for the patient explanation! This is an interesting argument that I'll have to think about some more, but I've already adjusted my view of how I expect things to go based on it.

Two questions:

First, isn't algorithmic trading a counterexample to your argument? It's true that it's a narrow domain, but it's also one where AI systems are trusted with enormous sums of money, and have the potential to make enormous losses. E.g. one company apparently lost $440 million in less than an hour due to a glitch in their software. Wikipedia on the consequences:

Knight Capital took a pre-tax loss of $440 million. This caused Knight Capital's stock price to collapse, sending shares lower by over 70% from before the announcement. The nature of the Knight Capital's unusual trading activity was described as a "technology breakdown".[14][15]

On Sunday, August 5 the company managed to raise around $400 million from half a dozen investors led by Jefferies in an attempt to stay in business after the trading error. Jefferies' CEO, Richard Handler and Executive Committee Chair Brian Friedman structured and led the rescue and Jefferies purchased $125 million of the $400 million investment and became Knight's largest shareholder. [2]. The financing would be in the form of convertible securities, bonds that turn into equity in the company at a fixed price in the future.[16]

The incident was embarrassing for Knight CEO Thomas Joyce, who was an outspoken critic of Nasdaq's handling of Facebook's IPO.[17] On the same day the company's stock plunged 33 percent, to $3.39; by the next day 75 percent of Knight's equity value had been erased.[18]

Also, you give several examples of AGIs potentially making large mistakes with large consequences, but couldn't e.g. a human strategist make a similarly big mistake as well?

You suggest that the corporate leadership could be held more responsible for a mistake by an AGI than if a human employer made the mistake, and I agree that this is definitely plausible. But I'm not sure whether it's inevitable. If the AGI was initially treated the way a junior human employee would, i.e. initially kept subject to more supervision and given more limited responsibilities, and then had its responsibilities scaled up as people came to trust it more and it learned from its mistakes, would that necessarily be considered irresponsible by the shareholders and insurers? (There's also the issue of privately held companies with no need to keep external shareholders satisfied.)

Right, I meant that Civ doesn't cheat when it comes to die rolls - e.g. if it displays a 75% chance for the player to win a battle, then the probability really is at least 75%.

It does cheat in a number of other ways.

I agree that it would be better not to have autonomously acting AIs, but not having any autonomously acting AIs would require a way to prevent anyone deploying them, and so far I haven't seen a proposal for that that'd seem even remotely feasible.

And if we can't stop them from being deployed, then deploying Friendly AIs first looks like the scenario that's more likely to work - which still isn't to say very likely, but at least it seems to have a chance of working even in principle. I don't see that an even-in-principle way for "just don't deploying autonomous AIs" to work.

I thought my excerpt answered that, but maybe that was illusion of transparency speaking. In particular, this paragraph:

In general, any broad domain involving high stakes, adversarial decision making and a need to act rapidly is likely to become increasingly dominated by autonomous systems. The extent to which the systems will need general intelligence will depend on the domain, but domains such as corporate management, fraud detection and warfare could plausibly make use of all the intelligence they can get. If oneʼs opponents in the domain are also using increasingly autonomous AI/AGI, there will be an arms race where one might have little choice but to give increasing amounts of control to AI/AGI systems.

To rephrase: the main trend is history has been to automate everything that can be automated, both to reduce costs and because machines can do things better than humans do. This isn't going to stop: I've already seen articles calling for both company middle managers, as well as government bureaucrats, to be replaced with AIs. If you have any kind of a business, you could potentially make it run better by putting a sufficiently sophisticated AI in charge - because it can think faster and smarter, deal with more information at once, and not have the issue of self-interest leading to office politics leading to many employees acting suboptimally from the company's point of view, that you'd get if you had a thousand human employees rather than a single AI.

This trend has been going on throughout history, doesn't show any signs of stopping, and inherently involves giving the AI systems whatever agency they need in order to run the company better.

And if your competitors are having AIs run their company and you don't, you're likely to be outcompeted, so you'll want to make sure your AIs are smarter and more capable of acting autonomously than the competitors. These pressures aren't just going to vanish at the point when AIs start approaching human capability.

The same considerations also apply to other domains than business - like governance - but the business and military domains are the most likely to have intense arms race dynamics going on.

This is too close to AGI for comfort IMO.

Pretty sure it's still comfortably narrow AI. People used to think that chess required AGI-levels of intelligence, too.

In MMORPGS, the game and setting are usually asymmetrical by design - there's no assumption that the human knight should have an equal amount of hit points as the ancient dragon, and it would actually violate the logic of the setting if that were the case.

The games where people do complain about AI cheating tend to put the enemies in a more symmetrical role - e.g. in something like Civilization or Starcraft, the game designers work to actively maintain an illusion that the AI players are basically just like human players and operating under the same rules.

If you break that illusion too blatantly, players will be reasonably annoyed, because they feel like the game is telling them one thing when the truth is actually different.

This may even have in-game ramifications: e.g. if I'm playing against a human opponent in a multiplayer match, I might want to keep my units hidden from him so that he doesn't know what I'm up to, but this is pointless against an AI opponent that sees the entire map all the time. (IIRC, in the original Red Alert, the Soviet player could construct buildings that recreated the shroud of war in areas that the enemy had already explored - and which were totally useless in single player, since the AI was never subject to the shroud of war!) In that case it's not just the player feeling cheated, it actively screws up the player's idea of what exactly would be a good idea against the AI.

May have been a vocal minority. You get some people incorrectly complaining about AI cheating in any game that utilizes randomness (Civilization and the new XCOMs are two examples I know of); usually this leads to somebody running a series of tests or decompiling the source code to show people that no, the die rolls are actually fair or (as is commonly the case) actually actively biased in the human player's favor.

This never stops some people from complaining nonetheless, but a lot of others find the evidence convincing enough and just chalk it up to their own biases (and are less likely to suspect cheating when they play the next game that has random elements).