About a year ago, I posted several ideas for "reduced impact AI" (what Nick Bostrom calls "domesticity"). I think the most promising approach was the third one, which I pompously titled "The information in the evidence". In this post, I'll attempt to put together a (non-realistic) example of this, to see if it's solid enough to build on. I'll be highlighting assumptions I'm making about the AI; please point out any implicit assumption that I missed, and any other weaknesses of the setup. For the moment, I'm more interested in "this doesn't work" than "this can't be done in practice" or "this can't be usefully generalised".

EDIT: It wasn't clear here, but any paperclip constructed by the reduced impact AI would be destroyed in the explosion, and the AIs would not be observed during the process. How to get useful work out of the AI will be the next step, if this model holds up.

Intuitive idea

For a reduced impact AI, we want an AI that can accomplish something, say building a paperclip, without it going out of control and optimising the universe. We want the future to be roughly the same whether or not the AI was turned on. Hence the piece of information "the AI was turned on" is not particularly important - if we didn't know, we wouldn't go far wrong in our predictions.

To enforce this we'll equip the AI with a two-piece motivation: a utility function U (causing it to build paperclips) and a penalty function R (which penalises the AI if its actions have a large future 'impact'). The challenge is to have a setup and a definition of R that implements this intuitive idea.

Reduced impact

Can we hope to define reduced impact? Doesn't any change percolate across all future time? Yes, but not all changes are the same - especially where our ability to predict them is concerned. Imagine an AI, confined to a box. While inside the box, it will build a paperclip. Then, in one hour's time, an antimatter bomb inside the box will explode, obliterating everything inside. We would intuitively say that whether the paperclip was built or not, would have very little impact afterwards. What we really mean is that we are unable to usefully use that information for future predictions. To make it even harder, surround the box by a large mass of antimatter gas, moving chaotically. In order to figure out the consequences of anything happening inside the box, we'd have to be able to unravel chaos: solve chaotic equation correctly and quantitatively, not just qualitatively. So our first assumption is that the AI can't unravel chaos.

A putative new idea for AI control; index here.

Thanks for help from Paul Christiano

If clippy, the paper-clip maximising AI, goes out of control, it would fill the universe with paper clips (or with better and better ways of counting the paper-clips it already has). If I sit down to a game with Deep Blue, then I know little about what will happen in the game, but I know it will end with me losing.

When facing a (general or narrow) superintelligent AI, the most relevant piece of information is what the AI's goals are. That's the general problem: there is no such thing as 'reduced impact' for such an AI. It doesn't matter who the next president of the United States is, if an AI wants to tile the universe with little smiley faces. But reduced impact is something we would dearly want to have - it gives us time to correct errors, perfect security systems, maybe even bootstrap our way to friendly AI from a non-friendly initial design. The most obvious path to coding reduced impact is to build a satisficer rather than a maximiser - but that proved unlikely to work.

But that ruthless maximising aspect of AIs may give us a way of quantifying 'reduced impact' - and hence including it in AI design. The central point being:

"When facing a (non-reduced impact) superintelligent AI, the AI's motivation is the most important fact we know."

Hence, conversely:

"If an AI has reduced impact, then knowing its motivation isn't particularly important. And a counterfactual world where the AI didn't exist, would not be very different from the one in which it does."

In this post, I'll be presenting some potential paths to formalising this intuition into something computable, giving us a numerical measure of impact that can be included in the AI's motivation to push it towards reduced impact. I'm putting this post up mainly to get help: does anyone know of already developed mathematical or computational tools that can be used to put these approaches on a rigorous footing?

Related to : Lost Purposes, The importance of Goodhart's Law, Homo Hypocritus, SIAI's scary idea, Value Deathism

Summary : Whenever human beings seek to achieve goals far beyond their individual ability, they use leverage of some kind of another. Creating organizations to achieve goals is a very powerful source of leverage. However due to their nature, organizations are imperfect levers and the primary purpose is often lost. The inertia of present forms and processes dominates beyond its useful period. The present system of the world has many such imperfect organizations in power and any of them developing near-general intelligence without significant redesign of their utility function can be a source of existential risk/values risk.

The eternally curious Tailsteak has written about how he always picks up pennies off the sidewalk. He's run a cost-benefit analysis and determined that it's better on average to pick up a penny than to pass it by. His mistake lies nowhere in the analysis itself; it's pretty much correct. His mistake is performing the analysis in the first place.

Pennies, you see, are easily the subject of scope insensitivity. When we come across a penny, we don't think, "Hey, that's something worth 0.05% of what I wish I had come across. I could buy a 25th of a gumball, a mouthful of an unhealthy carbonated beverage, a couple of seconds of footage on DVD, or enough gasoline to go a tenth of a mile." We think, "Hey, that's money," and we grab it.

The thing is, it's difficult to comprehend how little a penny is worth—we don't really have a separate concept for "mild happiness for a couple of seconds"—and we're likely to take risks that far outweigh the benefits. We don't think of bending over to pick up a penny as being a risky endeavour, but it's a penny. How much risk does it take to outweigh a penny? Surely the risk of "something unforeseen" easily does the job. Are you 99.999999% sure that picking up that penny won't kill you? You need a reason for every 9 (if you're ambivalent between using seven 9s and using nine 9s, you should use seven; the number of 9s is never arbitrary), and by the time you come up with eight reasons to pick up the penny, you'll have wasted several cents' worth of time. If you can reduce the probability of harm that far, I applaud you.

Of course, penny-grabbing doesn't have to involve actual pennies. Suppose that President Kodos of the Unified States of Somewhere (population 300 million) uses the word "idiot" in an important speech, causing the average citizen to scowl and ponder for one minute. Now, if a penny can buy you five seconds of happiness, and scowling and pondering brings the same amount of unhappiness, then that's twelve cents for every citizen, or 36 million dollars, of damage that Kodos just caused. Arguably, that's the value of a couple of human lives. As you can see, Kodos' decisions are extremely important. In this case, penny-grabbing would consist of anything less than trading precious seconds for precious human lives—if Kodos finds that he can save one life simply by going a few minutes out of his way, he should ignore it. (Photo ops and personal apologies are out of the question.) But keep in mind, of course, that avoiding saving someone's life because you have something better to do isn't rational unless you actually plan to do something better.

... Oh, no! I've been shot!

— C3PO

A strange sort of paralysis can occur when risk-averse people (like me) decide that we're going to play it safe. We imagine the worst thing that could happen if we go ahead with our slightly risky plan, and this stops us from carrying it out.

One possible way of overcoming such paralysis is to remind yourself just how much danger you're actually in.

Humanity could be mutilated by nuclear war, biotechnology disasters, societal meltdown, environmental collapse, oppressive governments, disagreeable AI, or other horrors. On an individual level, anybody's life could turn sour for more mundane reasons, from disease to bereavement to divorce to unemployment to depression. The terrifying scenarios depend on your values, and differ from person to person. Those here who hope to live forever may die of old age, and then cryonics turns out not to work.

There must be some number X which is the probability of Really Bad Things happening to you. X is probably not a tiny figure, but instead significantly above zero, which encourages you to go ahead with whatever slightly risky plan you were contemplating, as long as it only nudges X upwards a little.

Admittedly, this tactic seems like a cheap hack that relies on an error in human reasoning - is nudging your danger level from .2 to .201 actually more acceptable than nudging it from 0 to .001? Perhaps not. Needless to say, a real rationalist ought to ignore all this and take the action with the highest expected value.