That is, the first AI to go FOOM is so far superior in ability to anything else in the world that its subsequent steps of evolution are unconstrained by any outside pressures, and only follow either some sort of internal logic of value-change as intelligence increases, or else follow no logic at all, go in some sense on a "random walk" through possible values.
The AI is not supposed to change it values, regardless of whether it is powerful enough to realize them. Values are not up for grabs. Once the AI has some values it either wins and reshapes reality according to them or loses. Changing the values is one form losing. It seems that mostly anything that counts as a value system would object to changing an agent subscribing to that system into an agent using something else, so the AI won't follow any internal logic of value-change (unless some other agent forces it) and if it changes its values it will be by mistake (so closer to a random walk). Part of the idea of FAI is to build an AI that won't make those mistakes.
My own sense of how I create using my intelligence is that I try many different things. Many are tried purely in the sandbox of my own brain, run as simulations there, and only the more promising kept for further testing and development. It seems to me that my pool of ideas is an almost random noise of "what ifs" and that my creative intelligence is the discrimination function filtering which of these ideas are given more resources and which are killed in the crib.
The ideas coming into your awareness are very strongly pre-filtered; creativity is far from random noise. For one, the ideas are all relevant and somehow extrapolated from your knowledge of the world. Some of them might seem stupid but its only because of the pre-selection -- they never get compared to the idea of 'blue mesmerizingly up the slightly irreverent ladder, then dwarf the pegasus with the quantum sprocket' (and even this still makes a lot of sense compared to most random messages).
WHatever the next step, if its presence has the human race survive and thrive by doing the equivalent of what turned dinosaurs in to birds, or turned wolves into domesticated dogs, does that count as Friendly or Unfriendly?
It counts as failure to preserve humanity. An AI that does that is probably unfriendly (barring the coercion by external powerful agents. Eliezer actually wrote a story about such scenario, without AIs though.)
And is there really any point at all to fighting against it?
Sure seems like it.
It counts as failure to preserve humanity. An AI that does that is probably unfriendly (barring the coercion by external powerful agents. Eliezer actually wrote a story about such scenario, without AIs though.)
Interesting. I think I may even agree with you. In that story each race would need to conclude that the other races are "unfriendly". So Eliezer has written a story in which all the NATURAL intelligences (except us of course) are "unfriendly," and in which a human would need to agree that from the point of view of the other ...
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.