The AI might need a lot of localised information for friendliness, but it needn't be preprogrammed.

I don't know what you mean by 'preprogrammed',

I mean the proposal to solve morality and code it in to an AI.

and I don't know what view you think you're criticizing by making this point. MIRI generally supports indirect normativity, not direct normativity.

You have assumed that friendliness is a superset of morality.

A Friendly AI is, minimally, a situation-generally safe AGI.

Whrich is to say that full Fat friendliness is a superset of minimal friendliness . But minimal friendliness is just what I have been calling morality, and I dont see why I shouldn't continue. So friendliness is a superset of morality, as I said.

By the intelligence explosion thesis, 'situation-generally' will need to encompass the situation in which an AGI self-modifies to an ASI (artificial superintelligence), and since ASI are much more useful and dangerous than human-level AGIs, the bulk of the work in safety-proofing AGI will probably go into safety-proofing ASI.

.....by your assumptions, that morality/friendliness needs to say be solved separately from intelligence. But that is just what I am disputing.

A less minimal definition will say that Friendly AIs are AGIs that bring about situations humans strongly desire/value, and don't bring about situations they strongly dislike/disvalue. One could also treat this as an empirical claim about the more minimal definition: Any adequately safe AGI will be extremely domain-generally useful.

An AGI can be useful without wanting to do anything but answer questions accurately.

Regardless, nowhere in the above two paragraphs did I talk specifically about morality.

You didn't do use the word. But I think "not doing bad things, whilst not necessarily doing fun things either" picks out the same referent.

Moral values are important, but they are indeed a proper subset of human values, and we don't want an AGI to make everything worse forever even if it finds a way to do so without doing anything 'immoral'.

I find it hard to interpret that statement. How can making things worse forever not be immoral? What non-moral definition of worse are you mean using ?

That's like saying you can't simply folk physics down to real physics without sacrificing a lot of intuitions. Intuitions that are wrong need to go.

Getting rid of wrong intuitions may well make morality more complicated, rather than less. We agree that human folk morality may need to be refined a lot, but that gives us no reason to expect the task to be easy or the end-product to be simple

We have very good reason to think that the one true theory of something will be simpler, in Kolmogorov terms, than a mishmash of everybody's guesses Physics is simpler than folk physics. (It is harder to learn, because that requires the effortful system II to engage...but effort and complexity are different things).

And remember , my assumption is that the AI works out morality itself.

. Physical law appears to be simple, but it begets high-level regularities that are much less simple, like brains, genomes, and species. Morality occurs at a level closer to brains, genomes, and species than to physical law.

If an ASI can figure out such high level subjects as biology and decision theory, why shouldn't it be useful able to figure out morality?

But many problems would be insoluble under that assumption.

If human civilization depended on building an AI that can domain-generally speak English in a way that we'd ideally recognize as Correct, then I would be extremely worried. We can get away with shortcuts and approximations because speaking English correctly isn't very important. But getting small things permanently wrong about human values is important, when you're in control of the future of humanity.

Why wouldn't an AI that is smarter than US not be able to realise that for itself ?

It might not look fair that humans have to deal with such a huge problem, but the universe doesn't always give people reasonable-sized challenges.

If you assume that an AI is a learning system, then it can learn and does not need to be preprogrammed.

It has to be preprogrammed to learn the right things, and to incorporate the right things it's learned into its preferences.

That is confusingly phrased. A learning system needs some basis to learn, granted. You assume, tacitly, that it need not be preprogrammed with the right rules of grammar or economics. Why make exception for ethics?

Saying 'Just program the AI to learn the right preferences' doesn't solve the problem; programming the AI to learn the right preferences is the problem. See Detached lever fallacy:

A learning system needs some basis other than external stimulus to learn: given that, it is quite possible for most of the information to be contained in the stimulus, the data. Consider language. Do you think an AI will have to be preprogrammed with all the contents of every dictionary

"All this goes to explain why you can't create a kindly Artificial Intelligence by giving it nice parents and a kindly (yet occasionally strict) upbringing, the way it works with a human baby. As I've often heard proposed.

"It is a truism in evolutionary biology that conditional responses require more genetic complexity than unconditional responses. To develop a fur coat in response to cold weather requires more genetic complexity than developing a fur coat whether or not there is cold weather, because in the former case you also have to develop cold-weather sensors and wire them up to the fur coat.

"But this can lead to Lamarckian delusions: Look, I put the organism in a cold environment, and poof, it develops a fur coat! Genes? What genes? It's the cold that does it, obviously.

"There were, in fact, various slap-fights of this sort, in the history of evolutionary biology - cases where someone talked about an organismal response accelerating or bypassing evolution, without realizing that the conditional response was a complex adaptation of higher order than the actual response. (Developing a fur coat in response to cold weather, is strictly more complex than the final response, developing the fur coat.) [...]

"But the upshot is that if you have a little baby AI that is raised with loving and kindly (but occasionally strict) parents, you're pulling the levers that would, in a human, activate genetic machinery built in by millions of years of natural selection, and possibly produce a proper little human child. Though personality also plays a role, as billions of parents have found out in their due times.

"It's easier to program in unconditional niceness, than a response of niceness conditional on the AI being raised by kindly but strict parents. If you don't know how to do that, you certainly don't know how to create an AI that will conditionally respond to an environment of loving parents by growing up into a kindly superintelligence. If you have something that just maximizes the number of paperclips in its future light cone, and you raise it with loving parents, it's still going to come out as a paperclip maximizer. There is not that within it that would call forth the conditional response of a human child. Kindness is not sneezed into an AI by miraculous contagion from its programmers. Even if you wanted a conditional response, that conditionality is a fact you would have to deliberately choose about the design.

"Yes, there's certain information you have to get from the environment - but it's not sneezed in, it's not imprinted, it's not absorbed by magical contagion. Structuring that conditional response to the environment, so that the AI ends up in the desired state, is itself the major problem. 'Learning' far understates the difficulty of it - that sounds like the magic stuff is in the environment, and the difficulty is getting the magic stuff inside the AI. The real magic is in that structured, conditional response we trivialize as 'learning'. That's why building an AI isn't as easy as taking a computer, giving it a little baby body and trying to raise it in a human family. You would think that an unprogrammed computer, being ignorant, would be ready to learn; but the blank slate is a chimera."