Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Stuart_Armstrong comments on Domesticating reduced impact AIs - Less Wrong

8 Post author: Stuart_Armstrong 14 February 2013 04:59PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (104)

You are viewing a single comment's thread. Show more comments above.

Comment author: Strange7 18 February 2013 06:35:18AM 0 points [-]

I'm pretty sure this model is inherently a dead end for any useful applications. Even without gratuitous antimatter, a sufficiently smart AI trying to minimize it's future impact will put it's affairs in order and then self-destruct in some low-collateral-damage way which prevents anything interesting from being learned by analysis of the remains.

Comment author: Stuart_Armstrong 18 February 2013 12:58:52PM 1 point [-]

self-destruct in some low-collateral-damage way which prevents anything interesting from being learned by analysis of the remains

That's a plus, not a minus.

We can also use utility indifference (or something analogous) to get some useful info out.

Comment author: Strange7 21 February 2013 03:54:04PM 0 points [-]

It's a minus if you're trying to convince someone more results-oriented to keep giving you R&D funding. Imagine the budget meeting:

The EPA is breathing down our necks about venting a billion dollars worth of antimatter, you've learned literally nothing, and you consider that a good outcome?

If the AI is indifferent to future outcomes, what stops it from manipulating those outcomes in whatever way is convenient for it's other goals?

Comment author: Stuart_Armstrong 21 February 2013 04:44:38PM 0 points [-]

Indifference means that it cannot value any change to that particular outcome. More details at: http://www.fhi.ox.ac.uk/__data/assets/pdf_file/0020/18371/2010-1.pdf