The example still works since there are quite a few couples who use condoms because they just don't want to have kids. They don't have any worry about STDs from their partner. If you insist on a clear cut case look at men who get vasectomies.

A big part of the optimal strategy for present-day long-term reproductive success is to get rich, and a big part of getting rich is not having more kids than you can keep track of.

In terms of genetic success, having more kids than you can keep track of is pretty much the ideal, as long as all or at least most survive to reproductive adulthood.

In the modern context, if you impregnate someone without planning it out properly, there's a non-negligible chance they'll get an abortion, which is even worse for gene propagation. Furthermore, parents are to some extent legally responsible for their children's actions, so having too many poorly-regulated kids running around means exposing yourself to liability. A big part of the optimal strategy for present-day long-term reproductive success is to get rich, and a big part of getting rich is not having more kids than you can keep track of.

I think that's a retcon. People use contraception so they can have more sex than they would if they had to worry about having kids every time. They may or may not rationalise further, I suspect that generally they don't.

In the modern context, if you impregnate someone without planning it out properly, there's a non-negligible chance they'll get an abortion, which is even worse for gene propagation. Furthermore, parents are to some extent legally responsible for their children's actions, so having too many poorly-regulated kids running around means exposing yourself to liability. A big part of the optimal strategy for present-day long-term reproductive success is to get rich, and a big part of getting rich is not having more kids than you can keep track of.

But some people consciously choose never to have any kids. That's silly from the perspective of gene propagation if anything is.

Thus in trying to figure out what it's utility function actually is (like what humans are doing as they introspect more) it could discover that the only reason its goal is (something mysterious like) 'maximize paperclips' is because 'maximize paperclips' was how humans were (probabilistically inaccurately) expressing their preferences in some limited domain.

Hm. I suppose that's possible, though it would require that the AI be given a utility function that's specifically meant to be amenable to that kind of revision.

Under the most straightforward (i.e. not CEV-style) utility function design, fuzziness in its definition of "paperclip" would just drive the paperclip-maximizer to choose the possible definition that yields the highest utility score.

To pick a different silly example, a dog-maximizer with a utility function based on the number of dogs in the universe would simply prefer to tile the solar system with tiny Chihuahas rather than Great Danes; the whole range of "dog" definitions fit the function, so it just chooses the one that is most convenient for maximum utility. It wouldn't try to resolve it by trying to decide which definition is more in line with the designer's ideals, unless "consider the designer's ideals" were designed into the system from the start.

This is related to the theme Eliezer quite elegantly goes on about in Creating Friendly AI and that he for some reason barely mentioned in CEV, which is that the AI should look at its own source code as evidence of what its creators were trying to get at, and update its imperfect source code accordingly.

Yes, but it still has to be explicitly programmed to do that! The question is how to get it to do so. AFAIK shaper-anchor semantics is still quite a ways from being fully specified, but it seems the bigger obstacle is that an AI writer is less likely than not to take the effort to program it that way in the first place.

it's really damn hard to precisely specify a paperclip [...]

This is surely the kind of thing that superintelligences will be good at. They will have access to every paperclip picture on the net, every paperclip specification too. They will surely have a much clearer idea about what a paperclip is than humans do. They will know what boxes are too.

Right, but as far as I can tell without having put lots of hours into trying to solve the problem of clippyAI, it's really damn hard to precisely specify a paperclip.

I made a stab at it here, and it got some upvotes. So here's a repost:

Make a wire, 10 cm long and 1mm in diameter, composed of an alloy of 99.8% iron and 0.2% carbon. Start at one end and bend it such that the segments from 2-2.5cm, 2.75-3.25cm, 5.25-5.75cm form half-circles, with all the bends in the same direction and forming an inward spiral (the end with the first bend is outside the third bend).

(Please let me know if reposting violates LW ettiquette so I know not to do it again.)

Sure it does. A devout priest spends half his life celibate and serving God. One day he has a crisis, reads a bunch of stuff on the internet and suddenly realizes he doesn't believe in God. His values change.

Well, the counter-argument to that particular example would be that the priest's belief in God wasn't a terminal value; rather, their goals of being happy and helping other people and understanding the universe were. Believing in and obeying God were just instrumental values.

However, agreed that there's nothing in particular forcing people, weird and funky and clunky as our minds are, from always having the same fixed terminal values either. To pick an extreme example, peoples' brains can sometimes be messed up severely by hormonal imbalances, which can in turn cause people to do such drastically anti-own-terminal-value things as committing suicide.

I should've been more specific and just said that, in general, understanding evolutionary psychology never or only very rarely causes peoples' terminal values to change.

I suspect any concept of universal morality must be evolutionary. [...] We do value evolution in and of itself.

Human morality is a product of evolution; however, our morality is not itself an evolutionary algorithm execution mechanism. It's kind of a vague approximation of one (in that all the moralities that sucked for fitness were selected against), but it still often leads to drastically different results than a straight-up evolutionary fitness maximization algorithm with access to our brains' resources would.

For example: I intend never to have biological children and consider this decision to be a moral one. However, from an evolutionary perspective, deliberately preventing my own genes from propagating is just plain silly.

But just because such a thing is possible in principle doesn't make it the slightest bit likely.

Yes, the paper-clip maximizer is just a whimsical example. However, similarly Really Unfriendly optimizers are quite plausible. Imagine the horrors that could result from a naive human-happiness-maximizer hitting the singularity asymptote.

If you can build an AGI that can understand human language, it would be much easier and considerably more effective to make the AGI's goal system dynamically modifiable on reflection through human language.

Yes, that would be important, but it still wouldn't be enough to solve the problem; in fact, the really hard part of the problem still remains! The happiness-maximizer might base its understanding of happiness on descriptive human usage of the word, and end up with a truly thorough and consistent understanding of the word... and then still turn everybody into nearly mindless wireheads.

Our morality engines and our language aren't properly tuned for dealing with the kind of reality-bending power a superintelligent entity would have.

There's reason to suspect that any human-level AI must be programmed in human languages.

In fact, that's almost tautological by virtue of the Turing Test.

What?

Do you mean humanlike AIs? An AI capable of passing the Turing Test would of course need to understand human language well enough to act convincingly human (or at least do a really good imitation), but that's not necessarily a human-level AI (convincing people that you're human is a separate task from actually being human, probably a much easier one), and human-level AIs in general needn't necessarily understand human language any better than any other sort of language by default.

Anyway, an AI being "programmed in human languages" seems to be going by the "programming = instructions being given to a human servant" metaphor, and if you want that to work, you clearly first need to write the servant in something other than human language. And copying human psychology well enough that the AI actually understands human language as well as a human does, rather than being able to imitate understanding well enough to carry on a text-based conversation, is no easy task, and is probably a lot harder than manually coding a simple goal system like paperclip maximization in a lower-level language. But that could still be an AGI.

Programming an AI in C++? Doesn't compute.

Of the concepts named for Alan Turing it is "Turing Completeness" that is far more interesting and important than the chatbot test. If you think on the concept of a Turing complete computation system you will perhaps realise why the rest of us would consider your claim extremely silly. Well, one of the reasons anyway.

But it's a lot harder to code that than to code "maximize paperclips".

I'm claiming that the desire to code it in would be convergent, 'cuz it's the best way to do AI even if you think you're just trying to maximize paperclips.

Then you should have said that!

That sounds exactly like CEV.

Suppose someone who reliably does not generate common obviously wrong ideas/arguments has an uncommon idea that is wrong in a way that is non-obvious, but that you could explain if the wrong idea itself were precisely explained to you. But this person does not precisely explain their idea, but instead vaguely points to it with a description that sounds very much like a common obviously wrong idea. So you try to apply charity and fill in the gaps to figure out what they are really saying, but even if you do find the idea that they had in mind, you wouldn't identify as such, because you see how that idea is wrong, and being charitable, you can't interpret what they said in that way. How could you figure out what this person is talking about?

(Downvoted.)

You're responding to an interpretation of what I said that assumes I'm stupid [...] I'm not retarded.

No one said you were stupid.

Do you seriously think I've spent a year at SIAI without understanding such basic arguments?

People are responding to the text of your comments as written. If you write something that seems to ignore a standard argument, then it's not surprising that people will point out the standard argument.

As a parable, imagine an engineer proposing a design for a perpetual motion device. An onlooker objects: "But what about conservation of energy?" The engineer says: "Do you seriously think I spent four years at University without understanding such basic arguments?" An uncharitable onlooker might say "Yes." A better answer, I think, is: "Your personal credentials are not at issue, but the objection to your design remains."

I just don't have the energy to think through all the ways that people could interpret what I'm saying as something dumb because it pattern matches to things dumb people say. I'm going to start disclaiming this at the top of every comment, as suggested by Steve Rayhawk.

That sounds bad - perhaps reconsider.

Specifically, in this case, in the comment you replied to and elsewhere in this thread, I said: "this doesn't apply to AIs that are bad at that kind of philosophical reflection". I'm making a claim that all well-designed AIs will converge to universal 'morality' that we'd like upon reflection even if it wasn't explicitly coded to approximate human values. I'm not saying your average AI programmer can make an AI that does this, though I am suggesting it is plausible.

As a "peace offering", I'll describe a somewhat similar argument, although it stands as an open confusion, not so much a hypothesis.

Consider a human "prototype agent", additional data that you plug in into a proto-FAI (already a human-specific thingie, or course) to refer to precise human decision problem. Where does this human end, where are its boundaries? Why would its body be the cutoff point, why not include all of its causal past, all the way back to the Big Bang? At which point, talking about the human in particular seems to become useless, after all it's a tiny portion of all that data. But clearly we need to somehow point to the human decision problem, to distinguish it from frog decision problem and the like, even though such boundless prototype agents share all of their data. Do you point to human finger and specify this actuator as the locus through which the universe is to be interpreted, as opposed to pointing to a frog's leg? Possibly, but it'll take a better understanding of interpreting decision problems from arbitrary agents' definitions to make progress on questions like this.

You're responding to an interpretation of what I said that assumes I'm stupid, not the thing I was actually trying to say. Do you seriously think I've spent a year at SIAI without understanding such basic arguments?

With each comment like this you make, and lack of comments that show clear understanding, I think that more and more confidently, yes. Disclaimers don't help in such cases. You don't have to be stupid, you clearly aren't, but you seem to be using your intelligence to confuse yourself by lumping everything together instead of carefully examining distinct issues. Even if you actually understand something, adding a lot of noise over this understanding makes the overall model much less accurate.

Specifically, in this case, in the comment you replied to and elsewhere in this thread, I said: "this doesn't apply to AIs that are bad at that kind of philosophical reflection". I'm making a claim that all well-designed AIs will converge to universal 'morality' that we'd like upon reflection even if it wasn't explicitly coded to approximate human values.

One thing they rather obviously might converge on is the "goal system zero" / "Universal Instrumental Values" thing. The other main candidates seem to be "fitness" and "pleasure". These might well preserve humans for a while - in historical exhibits.

Does anyone know if Roko or Hollerith developed the idea much further?

Roko combined the conccept with the (rather less sensible) idea of promoting those instrumental values into terminal values - and was met with a chorus of "Unfriendly AI".

Hollerith produced several pages on the topic.

Probably the best-known continuation is via Omohundro.

"Universal Instrumental Values" is much the same idea as "Basic AI drives" dressed up a little differently:

• http://selfawaresystems.com/2007/11/30/paper-on-the-basic-ai-drives/

• http://selfawaresystems.com/2007/10/05/paper-on-the-nature-of-self-improving-artificial-intelligence/

Thus one the proposed titles: "Universal Instrumental Values".

Newsome didn't distinguish between instrumental and terminal values.