If it will help anyone, I'd like to chip in with a memory/note-taking technique I am using at the moment. Mindmaps. I find it extremely powerful for very fast information retrieval, since it's inherently hierarchical. I do digital mindmaps, using Freeplane. I use it for storing key ideas from books, articles, workflows, step by step howTos, programming snippets, even for my own thoughts. You name it.

Only downside I can think of is that my memory no longer has any incentive to retain the information I come across so you could say my memory only gets worse, using this technique. Does anyone know of any studies on long term effects on memory when storing information externally, rather than forcing your brain to do it itself?

This is like "X if 1 + 2 = 5". Not necessarily incorrect, but a bizarre statement. An agent with a single, non-reflective goal cannot want to change its goal. It may change its goal accidentally, or we may be incorrect about what its goals are, or something external may change its goal, or its goal will not change.

So it can change in the same way that you can decide right now that your only purposes will be torturing kittens and making giant cheesecakes. It can-as-reachable-node-in-planning do it, not can-as-physical-possibility. So it's possible to build entities with paperclip-maximizing or Friendly goals that will never in fact choose to alter them, just like it's possible for me to trust you won't enslave me into your cheesecake bakery.

I never said that paperclip maximizing can't be their ultimate life goal, but they could change it anytime they like.

This is incoherent. If X is my ultimate life goal, I never like to change that fact outside quite exceptional circumstances that become less likely with greater power (like "circumstances are such that X will be maximized if I am instead truly trying to maximize Y"). This is not to say that my goals will never change, but I will never want my "ultimate life goal" to change - that would run contrary to my goals.

it's impossible to create an AI (self-aware, learning) that has set values, that can't change

Why? Do you think paperclip maximizers are impossible?

whether to create AI only to perform certain simple tasks or whether to create a new race

You don't mean that as a dichotomy, do you?

(nods) Whether it's possible or not is generally an open question. There's a lot of skepticism about it (I'm fairly skeptical myself), but as with most technical questions, I'm generally content to have smart people research the question in more detail than I'm going to.

As to whether it's desirable, though... well, sure, of course it depends on our goals. If all I want is (as you say) to create a new race to replace humanity, and I'm indifferent as to the values of that race, then of course there's no reason for me to care about whether a self-improving AI I create will avoid value drift.

Personally, I'm more or less OK with something replacing humanity, but I'd prefer whatever that is to value certain things. For example, a commonly used trivial example around here of a hypothetical failure mode is a "paperclip maximizer" -- an AI that only valued the existence of paperclips, and consequently reassembled all matter it can get its effectors on as paperclips. A paperclip maximizer with powerful enough effectors reassembles everything into paperclips.

I would prefer that not happen, from which I conclude that I'm not in fact indifferent as to the values of a sufficiently powerful AI... I desire that such a system preserve at least certain values. (It is difficult to state precisely what values those are, of course. Human values are complex.) I therefore prefer that it avoid value drift with respect to those values.

How about you?

It is impossible for me to predict how a sufficiently complex system will react to most things. Heck, I can't even predict my dog's behavior most of the time. But there are certain things I know she values, and that means I can make certain predictions pretty confidently: she won't turn down a hot dog if I offer it, for example.

That's true more generally as well: knowing what a system values allows me to confidently make certain broad classes of predictions about it. If a superintelligent system wants me to suffer, for example, I can't predict what it's going to do, but I can confidently predict that I will suffer.

Burden of proof hasn't come up. I'm not trying to convince you of anything, I'm exploring your beliefs because I'm curious about them. (I'm similarly clarifying my beliefs when you ask about them.)

You might say that by introducing new desire (not to produce evil) I no longer desire (A), and I say, fine.

What I would actually say is that "don't produce evil" isn't a new value, and you didn't lose your original value ("intercourse") either. Rather, you started out with both values, and then you discovered that your values conflicted, and you chose to resolve that conflict by eliminating one of those values.

Presumably you eliminated your intercourse-value because it was the weaker of the two.. you valued it less. Had you valued intercourse more, you would instead have instead chosen to eliminate your desire to not be evil.

Another way of putting this is that you started out with two values which, aggregated, constituted a single complex value which is hard to describe in words.

Now, how do you want to ensure that the AI won't create it's own new desires based on new facts.

This is exactly right! The important trick is to build a system whose desires (I would say, rather, whose values) remain intact as it uncovers new facts about the world.

As you say, this is impossible if the system can derive values from facts... derive "ought" from "is." Conversely, it is theoretically possible, if facts and values are distinct sorts of things. So, yes: the goal is to build an AI architecture whose basic values are distinct from its data... whose "oughts" are derived from other "oughts" rather than entirely from "is"es.

You need to take desire out of the equation. The way you program the utility function fully determines the volition of the machine. It is the volition of the machine. Postulating that a machine can desire something that it's utility function doesn't define or include is roughly equivalent to postulating that 1 = 0. I think you might benefit from reading this actual SIAI article by Eliezer. It specifically address your concern.

There is one valid point - closely related to what you're saying here:

The AI would have access to every information humans ever conceived, including the discussions, disputes and research put into programming this AI's goals and nature. It may then adopt new goals based on the information gathered, realizing its former ones are no longer desirable.

But you're thinking about it the wrong way. The issue that the machine "realizes" that something is "no longer desirable" doesn't actually make a lot of sense because the AI is its programing and it can only "realize" things that its programing allows for (of course, since an AGI is so complicated, a simple utility function could result in a situation similar to presenting a Djinn (genie) an ill-specified request i.e. a be-careful-what-you-wish-for scenario).

A variant that does make sense and is a real concern is that as the AGI learns, it could change its definitions in unpredictable ways. Peter De Blanc talks about this here. This could lead to part of the utility function becoming undefined or to the machine valuing things that we never intended it to value - basically it makes the utility function unstable under the conditions you describe. The intuition is roughly that if you define a human in one way, according to what we currently know about physics, some new discovery made available to the AI might result in it redefining humans in new terms and no longer having them as a part of its utility function. Whatever the utility function describes is now separate from how humans appear to it.

are you saying that if you believe X, there can't possibly exist any information that you haven't discovered yet that could convince your belief is false?

No. I'm saying that if I value X, I can't think of any information that would cause me to value NOT(X) instead.

Can you give me an example of something you desire not to do, which you would willingly edit yourself to desire to do?