How good at playing chess would a chess computer have to be before it started trying to feed the hungry?

I think you won't find a very good argument either way, because different ways of building AIs create different constraints on the possible motivations they could have, and we don't know which methods are likely to succeed (or come first) at this point.

For example, uploads would be constrained to have motivations similar to existing humans (plus random drifts or corruptions of such). It seems impossible to create an upload who is motivated solely to fill the universe with paperclips. AIs created by genetic algorithms might be constrained to have certain motivations, which would probably differ from the set of possible motivations of AIs created by simulated biological evolution, etc.

The Orthogonality Thesis (or it's denial) must assume that certain types of AI, e.g., those based on generic optimization algorithms that can accept a wide range of objective functions, are feasible (or not) to build, but I don't think we can safely make such assumptions yet.

ETA: Just noticed Will Newsome's comment, which makes similar points.

A possible intuition may come from computer games. In principle, a game world can be arbitrarily complex, and the goals ("winning conditions") are specified by the programmers, so they can be anything at all. For example, the world may be Medieval Fantasy setting, while the goal may be to invent, craft, take from NPCs by force, and otherwise collect as many paperclips as possible. If an external incentive is provided for people to play the game (e.g., they get real-world money for each paperclip), the players will essentially become paperclip maximizers within the game world.

I've had several conversations that went like this:

Victim: But surely a smart artificial intelligence will be able to tell right from wrong, if we humans can do that?

Me: Forget about the word "intelligence" for a moment. Imagine a machine that looks at all actions in turn, and mechanically chooses the action that leads to producing the greatest number of paperclips, in whichever way possible. With enough computing power and enough knowledge about the outside world, the machine might find a way to convert the whole world into a paperclip factory. The machine will resist any attempts by humans to interfere, because the machine's goal function doesn't say anything about humans, only paperclips.

Victim: But such a machine would not be truly intelligent.

Me: Who cares about definitions of words? Humanity can someday find a way to build such a machine, and then we're all screwed.

Victim: ...okay, I see your point. Your machine is not intelligent, but it can be very dangerous because it's super-efficient.

Me (under my breath): Yeah. That's actually my definition of "superintelligent", but you seem to have a concept of "intelligence" that's entangled with many accidental facts about humans, so let's not go there.

Accept that moral conceptual truths are possible, and instead argue that an AI would deliberately try to not learn them.

What distinguishes the "Orthogonality thesis" from "Hume's Guillotine"? If you're looking for standard published arguments, I'd think you could start with "A Treatise of Human Nature" and proceed through the history of the "is-ought problem" from there.

As an example of a fairly powerful optimization process with very unhuman goals, you can cite evolution, which is superhuman in some ways, yet quite amoral.

Possibly somewhat off-topic: my hunch is that the actual motivation of the initial AGI will be random, rather than orthogonal to anything.

Consider this: how often has a difficult task been accomplished right the first time, even with all the careful preparation beforehand? For example, how many rockets blew up, killing people in the process, before the first successful lift-off? People were careless but lucky with the first nuclear reactor, though note "Fermi had convinced Arthur Compton that his calculations were reliable enough to rule out a runaway chain reaction or an explosion, but, as the official historians of the Atomic Energy Commission later noted, the "gamble" remained in conducting "a possibly catastrophic experiment in one of the most densely populated areas of the nation!"

I doubt that one can count on luck in the AGI development, but I would bet on unintentional carelessness (and other manifestations of the Murphy's law).

The bottom line is (nothing new here), no matter how much you research things beforehand, the first AGI will have bugs, with unpredictable consequences for its actual motivation. If we are lucky, there will be a chance to fix the bugs. Whether it is even possible to constrain the severity of bugs is way too early to tell, given how little is currently known about the topic.

When I looked at the puppy, I realized this:

At the moment when you create the AIs, their motivation and intelligence could be independent. But if let them run for a while, some motivations will lead to changes in intelligence. Improving intelligence could be difficult, but I think it is obvious that motivation to self-destruct will on average decrease the intelligence.

So are you talking about orthogonality of motivation and intelligence in freshly created AIs, or in running AIs?

It isn't a definitive argument, but you could point out that various intelligent historical figures had different morals from modern intelligent people. Napoleon, for instance--his intelligence is apparent, but his morality is ambiguous. Newton, or Archimedes, or George Washington, or any of several others, would work similarly.

A lot of the arguments given in these comments amount to: We can imagine a narrow AI that somehow becomes a general intelligence without wireheading or goal distortion, or, We can imagine a specific AGI architecture that is amenable to having precisely defined goals, and because we can imagine them, they're probably possible, and if they're probably possible, then they're probable. But such an argument is very weak. Our intuitions might be wrong, those AIs might not be the first to be developed, they might be theoretically possible but not pragmatically possible, and so on. Remember, we still don't know what intelligence is! We can define it as cross-domain optimization or what have you, but such definitions are not automatically valid just because they look sorta math-y. AIXI is probably not intelligent in the sense that a human is intelligent, and thus won't be dangerous. Why should I believe that any other AI architectures you come up with on the fly are any more dangerous?

So whenever you say, "imagine an AIXI approximation with a specific non-friendly utility function: that would be bad!", my response is, "who says such an AGI is even possible, let alone probable?". And whenever you say, "Omohundro says...", my response is, "Omohundro's arguments are informal and suggestive, but simply nowhere near conclusive, and in fact parts of his arguments can be taken to suggest in favor of an AI detecting and following moral law". There just aren't any knock-down arguments, because we don't know what it takes to make AGI. The best you can do is to make pragmatic arguments that caution is a good idea because the stakes are high. When people in this community act as if they have knock-down arguments where there aren't any it makes SingInst and LessWrong look like weirdly overconfident end-of-the-world-mongers.

(Also, the 'AGI will literally kill us all by default' argument is laughably bad, for many game theoretic and economic reasons both standard and acausal that should be obvious, and people unthinkingly repeating it also makes SingInst and LessWrong look like weirdly overconfident end-of-the-world-mongers.)

http://philosophicaldisquisitions.blogspot.com/2012/04/bostrom-on-superintelligence-and.html

Sure, utility and intelligence might be orthogonal. But very different utilities could still lead to very similar behaviors.

Assuming, from the title, that you're looking for argument by counterexample...

The obvious reply would be to invoke Godwin's Law - there's a quote in Mein Kampf along the lines of "I am convinced that by fighting off the Jews, I am doing the work of our creator...". Comments like this pretty reliably generate a response something like "Hitler was a diseased mind/insane/evil!" to which you may reply "Yeah, but he was pretty sharp, too." However, this has the downside of invoking Nazis, which in a certain kind of person may provoke an instant "This is a reactionary idiot" response and a complete discarding of the argument. So it's a temperamental trick, and I'm not skilled enough in the dark arts to know if it's a net gain.

On the other hand, you might prefer Pol Pat, or Ted Bundy, or any of a very large number of dictators and serial killers who don't produce the same mindkilling response as Hitler.

A lot of fictional evidence comes to mind as well, but we do try not to generalize from that... Still, if you just want to WIN the argument rather than win rationally, it may help to pull an example from some media form that the audience is likely to appreciate. Lex Luthor, Snidely Whiplash, Yagami Light (or L, if you prefer), Mephistopheles (or Faust), and so on.

Is that the sort of thing you wanted?

One of the most annoying arguments when discussing AI is the perennial "But if the AI is so smart, why won't it figure out the right thing to do anyway?" It's often the ultimate curiosity stopper.

How is this a curiosity stopper? It's a good question, as is evidenced by your trying to find an answer to it.

(Note that I do not necessarily agree with what I wrote below. You asked for possible counter-arguments. So here goes.)

Might intelligence imply benevolence?

I believe that a fundamental requirement for any rational agent is the motivation to act maximally intelligently and correctly. That requirement seems even more obvious if we are talking about a conjectured artificial general intelligence (AGI) that is able to improve itself to the point where it is substantially better at most activities than humans. Since if it wouldn't want to be maximally correct then it wouldn't become superhuman intelligent in the first place.

If we consider giving such an AGI a simple goal, e.g. the established goal of paperclip maximization. Is it really clear that human values are not implicit even given such a simplistic goal?

To pose an existential risk in the first place, an AGI would have to maximize paperclips in an unbounded way, eventually taking over the whole universe and convert all matter into paperclips. Given that no sane human would explicitly define such a goal, an AGI with the goal of maximizing paperclips would have to infer it as implicit to do so. But would such an inference make sense, given its superhuman intelligence?

The question boils down to how an AGI would interpret any vagueness present in its goal architecture and how it would deal with the implied invisible.

Given that any rational agent, especially AGI's capable of recursive self-improvement, want to act in the most intelligent and correct way possible, it seems reasonable that it would interpret any vagueness in a way that most closely reflects the most probable way it was meant to be interpreted.

Would it be intelligent and correct to ignore human volition in the context of maximizing paperclips? Would it be less wrong to maximize paperclips in the most literal sense possible?

The argument uttered by advocates of friendly AI is that any AGI that isn't explicitly designed to be friendly won't be friendly. But I wonder how much sense this actually makes.

Every human craftsman who enters into an agreement is bound by a contract that includes a lot of implied conditions. Humans use their intelligence to fill the gaps. For example, if a human craftsman is told to decorate a house, they are not going to attempt to take over the neighbourhood to protect their work.

A human craftsman wouldn't do that, not because they share human values, but simply because it wouldn't be sensible to do so given the implicit frame of reference of their contract. The contract implicitly includes the volition of the person that told them to decorate their house. They might not even like the way they are supposed to do it. It would simply be stupid to do it any different way.

How would a superhuman AI not contemplate its own drives and interpret them given the right frame of reference, i.e. human volition? Why would a superhuman general intelligence misunderstand what is meant by "maximize paperclips", while any human intelligence will be better able to infer the correct interpretation?

My argument: The reason the orthogonal hypothesis is false is that 100% (full) general intelligence is not possible without 100% (perfect) moral motivations. That is to say, any AI that is not completely moral is limited to some degree.

Expert Counter-argument: But the Marcus Hutter AIXI model shows that it’s possible to have optimal (fully general intelligence) decision makers with any arbitrary goals.

My reply: The AIXI model of Hutter is not a true general intelligence, because the initial goals are fixed (i.e it cannot reflect upon and modify its own goals). Further, the AIXI model is not pratical in real-time due to requirements for infinite (or impractically high) computational resources. You need correct priors for accurate real-time reasoning.

Expert Counter-argument: What have priors got to do with morality? Further, the problem of priors has been solved. The Kolmogorov complexity prior is a precise formalization of Occam’s razor.

My reply: The Kolmogorov complexity prior is uncomputable

Expert Counter-argument: There are perfectly workable approximations! For instance, the Schmidhuber speed prior and methods such as Monte Carlo methods can be deployed.

My reply: There is no proof that such approximations can be scaled up from their success in limited domains to full general intelligence.

Expert Counter-argument: The problem of priors isn’t relevant. The initial priors ‘wash out’ with enough evidence. The convergence theorems show this.

My reply: In fact, there is no convergence for different initial Bayesian models. Further, the computational explosion with the increase in problem complexity results in failure of Occam approximations to scale.

Expert Counter-argument: Even if this is true (which is highly debatable), what have priors got to do with morality any way?

My reply: Occam’s razor is the smoking gun which establishes a link between priors and value judgements. Fully general intelligence requires a formal notion of Occam razor. But Occam priors are equivalent to an aesthetic notion of what constitutes ‘simplicity’. Schmidhuber argued that the notion of ‘beauty’ is tied into the ability to compress information. But if this is true, then intelligence cannot be independent of aesthetic/value judgements.

Expert Counter-argument: This sounds like a weak idea to me. I don’t see the link between aesthetic judgements and Occam priors. As to Schmidhuber’s attempt to tie a notion of ‘beauty’ to data compression, didn’t Rayhawk critique the idea on the grounds that Schmidhuber’s notion of beauty isn’t actually required for an optimal decision making agent? And even if the general idea was correct, why should this notion of ‘beauty’ correspond to what humans value any way?

My reply: Occam priors could be approximated by deploying categorization and analogical inference...grouping things into categories to compress information is equivalent to the introduction of inductive biases, which already looks human-like. Kant argued that intelligent minds require a set of universal a-priori categories and ontological primitives look like the modern version of Kant’s categories.

Expert Counter-argument: This is just vague hand-waving! Exactly how would your analogical inference/categorization scheme work?

My reply: A small set of initial ontological primitives (defined in terms of prototypes rather than sufficient conditions) would be built-into the seed-AI. These would serve as reference classes enabling calculation of the semantic distances between features of concepts in feature-space. This would define the basic notion of ‘similarity’ needed for analogical inference, which would involve cross-domain mappings between ontologies and allow fully general (cross-domain) reasoning.

Expert Counter-argument: OK, although I’m not sure you know what you’re talking about, and I still don’t see the connection to morality

My reply: Here's the connection to morality: the ontology would be used to form representations of our values in the form of decision-making narratives. To compress the representations of values into these coherent narratives, aesthetic and ethical value judgments would be needed. This whole system would be defined in terms of information theory.

Expert Counter-argument: Even if there was a ‘natural’ notion of morality that emerges from all this, which sounds just like wild speculation to me, why should I (or an AI) want to follow it anyway?

My reply: Deviations from correct compression techniques degrade representations of decision-making narratives, thus degrading general reasoning abilities. Since correct compression techniques are tied to aesthetic/value judgements, any deviation from perfect morality would reduce the effective intelligence of the AI. You cannot reason that you should deviate from perfect morality without contradicting yourself and degrading your own cognitive processes! So knowledge of universal morality would automatically be self-motivating.

Expert Counter-argument: Sounds unlikely to me, besides, as Bostrom pointed out, even if there was a universal morality, the knowledge of which is self-motivating, the AI need not be built in the way you described, in which case it wouldn’t care about correct reasoning as you have defined it. Pick a random mind from mind-space, it doesn’t care about humans.

My reply: Most programs picked randomly from mind-space don’t work. UAIs would not work properly. Most of the AIs picked randomly from mind-space just won’t work. I’m not claiming intelligence implies morality, I’m claiming the converse of this (i.e ‘morality implies intelligence’). I’m not saying UAIs can’t exist, I’m saying they won’t be true general intelligences. I’m saying they won’t be able to self-improve to the super-intelligent level.

Expert: Let me think this further for while.....

Isn't acting maximally intelligently and correctly itself a motivation? The question you are really asking seems to be why an AI is supposed to act maximally intelligently and correctly to achieve world states that are not explicitly or implicitly defined to maximize expected utility. Yet the motivation to act maximally intelligently and correctly will always be given, otherwise you're not talking about an rational agent.

I'm hoping to complement this with a paper laying out the positive arguments in favour of the thesis. So I'm asking you for your strongest arguments for (or against) the orthogonality thesis.

Any kind of agent could - in principle - be engineered.

However, some sorts of agent are more likely to evolve than others - and it is this case that actually matters to us.