" How is it that the AGI is yet smart enough to learn this all by itself but fails to notice that there are rules to follow" - because there is no reason for an AGI automagically creating arbitrary restrictions if they aren't part of the goal or superior to the goal. For example, I'm quite sure that F1 rules prohibit interfering with drivers during the game; but if somehow a silicon-reaction-speed AGI can't win F1 by default, then it may find it simpler/quicker to harm the opponents in one of the infinity ways that the F1 rules don't cover - say, getting some funds in financial arbitrage, buying out the other teams, and firing any good drivers or engineering a virus that halves the reaction speed of all homo-sapiens - and then it would be happy as the goal is achieved within the rules.

One argument is that the only general intelligence we know, humans, would want to improve if they could tinker with their source code. But why is it so hard to make people learn then? Why don't we see much more people interested in how to change their mind?

Humans are (roughly) the stupidest possible general intelligences. If it were possible for even a slightly less intelligent species to have dominated the earth, they would have done so (and would now be debating AI development in a slightly less sophisticated way). We are so amazingly stupid we don't even know what our own preferences are! We (currently) can't improve or modify our hardware. We can modify our own software, but only to a very limited extent and within narrow constraints. Our entire cognitive architecture was built by piling barely-good-enough hacks on top of each other, with no foresight, no architecture, and no comments in the code.

And despite all that, we humans have reshaped the world to our whims, causing great devastation and wiping out many species that are only marginally dumber than we are. And no human who has ever lived has known their own utility function. That alone would make us massively more powerful optimizers; it's a standard feature for every AI. AIs have no physical, emotional, or social needs. They do not sleep, or rest, or get bored or distracted. On current hardware, they can perform more serial operations per second than a human by a factor of 10,000,000.

An AI that gets even a little bit smarter than a human will out-optimize us, recursive self-improvement or not. It will get whatever it has been programmed to want, and it will devote every possible resource it can acquire to doing so.

But in my opinion the difference here also implies that while nothing will distract them, there will also be no incentive not to hold. Why would it do more than necessary to reach a goal?

Clippy's cousin, Clip, is a paperclip satisficer. Clip has been programmed to create 100 paperclips. Unfortunately, the code for his utility function is approximately "ensure that there are 100 more paperclips in the universe than there were when I began running."

Soon, our solar system is replaced with n+100 paperclips surrounded by the most sophisticated defenses Clip can devise. Probes are sent out to destroy any entity that could ever have even the slightest chance of leading to the destruction of a single paperclip.

The further argument here is that it will misunderstand its goals. But the problem I see in this case is firstly that the more unspecific the goal the less it is able to measure its self-improvement against the goal to quantify the efficiency of its output.

The Hidden Complexity of Wishes and Failed Utopia #4-2 may be worth a look. The problem isn't a lack of specificity, because an AI without a well-defined goal function won't function. Rather, the danger is that the goal system we specify will have unintended consequences.

Secondly, the more vague a goal the larger has to be its general knowledge, previous to any self-improvement, to make sense of it in the first place? Shouldn't those problems outweigh each other to some extent?

Of course, if you told it to learn as much about the universe as possible, that is something completely different.

Acquiring information is useful for just about every goal. When there aren't bigger expected marginal gains elsewhere, information gathering is better than nothing. "Learn as much about the universe as possible" is another standard feature for expected utility maximizers.

And this is all before taking into account self-improvement, utility functions that are unstable under self-modification, and our dear friend FOOM.

TL;DR:

• Agents that aren't made of meat will actually maximize utility.
• Writing a utility function that actually says what you think it does is much harder than it looks.
• Be afraid.

Yes, but each step into that direction does also provide insights into the nature of AI and therefore can help to design friendly AI.

This was my guess for the first 1.5 years or so. The problem is, FAI is necessarily a strong AGI, but if you learn how to build a strong AGI, you are in trouble. You don't want to have that knowledge around, unless you know where to get the goals from, and studying efficient AGIs doesn't help with that. The harm is greater than the benefit, and it's entirely plausible that one can succeed in building a strong AGI without getting the slightest clue about how to define Friendly goal, so it's not a given that there is any benefit whatsoever.

If we are talking about a full-fledged general intelligence here (Skynet), there's no arguing against any risk. I believe all we disagree about are definitions. That there are risks from advanced real-world (fictional) nanotechnology is indisputable. I'm merely saying that what researchers are working on is nanotechnology with the potential to lead to grey goo scenarios but that there is no inherent risk that any work on it will lead down the same pathway.

Please keep focus, which is one of the most important tools. The above paragraph is unrelated to what I addressed in this conversation.

It is incredible hard to come up with an intelligence that knows what planning consists of and to know and care to be able to judge what step is instrumental. This won't just happen accidentally and will likely necessitate knowledge sufficient to be able to set scope boundaries as well.

Review the above paragraph: what you are saying is that AIs are hard to build. But of course chess AIs do plan, to give an example. They don't perform only the moves they are "told" to perform.