virtually nothing

OK, but does anything survive? How about the idea that

• Some systems will be opaque to human programmers

• ...they will also be opaque to themselves

• ..which will stymie recursive self-improvement.

The biggest problem here is that you start from the assumption that current neural net systems will eventually be made into AI systems with all the failings and limitations they have now. You extrapolate massively from the assumption.

But there is absolutely no reason to believe that the evolutionary changes to NN that are required in order to make them fully intelligent (AGI) will leave them with the all the same characteristics they have now. There will be SO MANY changes, that virtually nothing about the current systems will be true of those future systems.

Which renders your entire extrapolation moot.

The biggest problem here is that you start from the assumption that current neural net systems will eventually be made into AI systems with all the failings and limitations they have now. You extrapolate massively from the assumption.

But there is absolutely no reason to believe that the evolutionary changes to NN that are required in order to make them fully intelligent (AGI) will leave them with the all the same characteristics they have now. There will be SO MANY changes, that virtually nothing about the current systems will be true of those future systems.

Which renders your entire extrapolation moot.

Excuse me, but as much as I think the SIAI bunch were being rude to you, if you had presented, at a serious conference on a serious topic, a paper that waves its hands, yells "Complexity! Irreducible! Parallel!" and expected a good reception, I would have been privately snarking if not publicly. That would be me acting like a straight-up asshole, but it would also be because you never try to understand a phenomenon by declaring it un-understandable. Which is not to say that symbolic, theorem-prover, "Pure Maths are Pure Reason which will create Pure Intelligence" approaches are very good either -- they totally failed to predict that the brain is a universal learning machine, for instance.

(And so far, the "HEY NEURAL NETS LEARN WELL" approach is failing to predict a few things I think they really ought to be able to see, and endeavor to show.)

That anyone would ever try to claim a technological revolution is about to arise from either of those schools of work is what constantly discredits the field of artificial intelligence as a hype-driven fraud!

Following some somewhat misleading articles quoting me, I thought I’d present the top 10 myths about the AI risk thesis:

1) That we’re certain AI will doom us. Certainly not. It’s very hard to be certain of anything involving a technology that doesn’t exist; we’re just claiming that the probability of AI going bad isn’t low enough that we can ignore it.

MISLEADING.

If by "we" you mean the people who have published their thoughts on this matter. I believe I am right in saying that you have in the past referenced Steve Omohundro's paper, in which he says:

Without special precautions, [the AGI] will resist being turned off, will try to break into other machines and make copies of itself, and will try to acquire resources without regard for anyone else’s safety. These potentially harmful behaviors will occur not because they were programmed in at the start, but because of the intrinsic nature of goal driven systems (Omohundro, 2008).

Although this begins with "without special precautions" it then goes on to ONLY list all the ways in which this could happen, with no suggestions about how "special precautions" are even possible.

This cannot be construed as "we’re just claiming that the probability of AI going bad isn’t low enough that we can ignore it." The quote is also inconsistent with your statement "It’s very hard to be certain of anything involving a technology that doesn’t exist", because Omohundro says categorically that this "will occur ... because of the intrinsic nature of goal driven systems".

I picked Omohundro's paper as an example, but there are numerous similar writings from MIRI and FHI. I listed several examples in my AAAI paper (Loosemore, 2014).

2) That humanity will survive, because we’ve always survived before. Many groups of humans haven’t survived contact with more powerful intelligent agents. In the past, those agents were other humans; but they need not be. The universe does not owe us a destiny. In the future, something will survive; it need not be us.

STRAWMAN.

I haven't seen anyone of significance make that claim, so how can it be a "myth about the AI risk thesis"?

3) That uncertainty means that you’re safe. If you’re claiming that AI is impossible, or that it will take countless decades, or that it’ll be safe... you’re not being uncertain, you’re being extremely specific about the future. “No AI risk” is certain; “Possible AI risk” is where we stand.

STRAWMAN.

Again, only a tiny minority have said anything resembling those claims, so how can they be a "myth about the AI risk thesis"?

4) That Terminator robots will be involved. Please? The threat from AI comes from its potential intelligence, not from its ability to clank around slowly with an Austrian accent.

STRAWMAN.

Journalists and bloggers love to put a Terminator picture on their post as an Eyeball Magnet. Why elevate an Eyeball Magnet to the level of a "myth about the AI risk thesis"?

5) That we’re assuming the AI is too dumb to know what we’re asking it. No. A powerful AI will know what we meant to program it to do. But why should it care? And if we could figure out how to program “care about what we meant to ask”, well, then we’d have safe AI.

WRONG.

I published a paper giving a thorough analysis anddebunking of your (MIRI and FHI) claims in that regard -- and yet neither you nor anyone else at MIRI or FHI has ever addressed that analysis. Instead, you simply repeat the nonsense as if no one has ever refuted it. MIRI was also invited to respond when the paper was presented. They refused the invitation.

6) That there’s one simple trick that can solve the whole problem. Many people have proposed that one trick. Some of them could even help (see Holden’s tool AI idea). None of them reduce the risk enough to relax – and many of the tricks contradict each other (you can’t design an AI that’s both a tool and socialising with humans!).

INCOHERENT.

I have heard almost no one propose that there is "one simple trick", so how can it be a "myth" (you need more than a couple of suggestions, for something to be a myth).

More importantly, what happened to your own Point 1, above? You said "It’s very hard to be certain of anything involving a technology that doesn’t exist" -- but now you are making categorical statements (e.g. "None of them reduce the risk enough to relax" and "you can’t design an AI that’s both a tool and socialising with humans!") about the effectiveness of various ideas about that technology that doesn’t exist.

7) That we want to stop AI research. We don’t. Current AI research is very far from the risky areas and abilities. And it’s risk aware AI researchers that are most likely to figure out how to make safe AI.

PARTLY TRUE

......... except for all the discussion at MIRI and FHI regarding Hard Takeoff scenarios. And, again, whence cometh the certainty in the statement "Current AI research is very far from the risky areas and abilities."?

8) That AIs will be more intelligent than us, hence more moral. It’s pretty clear than in humans, high intelligence is no guarantee of morality. Are you really willing to bet the whole future of humanity on the idea that AIs might be different? That in the billions of possible minds out there, there is none that is both dangerous and very intelligent?

MISLEADING, and a STRAWMAN.

Few if any people have made the claim that increased intelligence BY ITSELF guarantees greater morality.

This is misleading because some people have discussed a tendency (not a guarantee) for higher intelligence to lead to greater morality (Mark Waser's papers go into this in some detail). Combining that with the probability of AI going through a singleton bottleneck, and there is a plausible scenario in which AIs themselves enforce a post-singleton constraint on the morality of future systems.

You are also profoundly confused (or naive) about how AI works, when you ask the question "Are you really willing to bet the whole future of humanity on the idea that AIs might be different?" One does not WAIT to find out if the motivation system of a future AI "is different", one DESIGNS the motivation system of a future AI to be either this way or that way.

It could be that an absolute correlation between increased intelligence and increased morality, in humans, is undermined by the existence of a psychopathic-selfish module in the human motivation system. Solution? Remove the module. Not possible to do in humans because of the biology, but trivially easy to do if you are designing an AI along the same lines. And if this IS what is happening in humans, then you can deduce nothing about future AI systems from the observation that "in humans, high intelligence is no guarantee of morality".

9) That science fiction or spiritual ideas are useful ways of understanding AI risk. Science fiction and spirituality are full of human concepts, created by humans, for humans, to communicate human ideas. They need not apply to AI at all, as these could be minds far removed from human concepts, possibly without a body, possibly with no emotions or consciousness, possibly with many new emotions and a different type of consciousness, etc... Anthropomorphising the AIs could lead us completely astray.

MISLEADING and CONFUSED.

This is a confusing mishmash of speculation and assumption.

A sentence like "minds far removed from human concepts" is not grounded in any coherent theory of what a 'mind' is or what a 'concept' is, or how to do comparative measures across minds and concepts. The sentence is vague, science-fictional handwaving.

The same goes for other statements like that the AI might have "no emotions or consciousness". Until you define what you mean by those terms, and give some kind of argument about why the AI would or would not be expected to have them, and what difference it would make, in either case, the statement is just folk psychology dressed up as science.

Lists cannot be comprehensive, but they can adapt and grow, adding more important points: 1) That AIs have to be evil to be dangerous. The majority of the risk comes from indifferent or partially nice AIs. Those that have sone goal to follow, with humanity and its desires just getting in the way – using resources, trying to oppose it, or just not being perfectly efficient for its goal.

MISLEADING and a STRAWMAN.

Yet again, I demonstrated in my 2014 paper that that claim is incoherent. It is predicated on a trivially stupid AI design, and there is no evidence that such a design will ever work in the real world.

If you, or anyone else at MIRI or FHI think that you can answer the demolition of this idea that I presented in the AAAI paper, it is about time you published it.

2) That we believe AI is coming soon. It might; it might not. Even if AI is known to be in the distant future (which isn't known, currently), some of the groundwork is worth laying now).

ACCURATE.

References

Loosemore, R.P.W. (2014). The Maverick Nanny with a Dopamine Drip: Debunking Fallacies in the Theory of AI Motivation. Association for the Advancement of Artificial Intelligence 2014 Spring Symposium, Stanford, CA.

Omohundro, Stephen M. 2008. The Basic AI Drives. In Wang, P., Goertzel, B. and Franklin, S. (Eds), Artificial General Intelligence 2008: Proceedings of the First AGI Conference. Amsterdam: IOS.

Wait wait wait. You didn't head to the dinner, drink some fine wine, and start raucously debating the same issue over again?

Bah, humbug!

Also, how do I get invited to these conferences again ;-)?

It is a scandalously unjustified assumption, made very hard to attack by the fact that it is repeated so frequently that everyone believes it be true just because everyone else believes it.

Very true, at least regarding AI. Personally, my theory is that the brain does do reinforcement learning, but the "reward function" isn't a VNM-rational utility function, it's just something the body signals to the brain to say, "Hey, that world-state was great!" I can't imagine that Nature used something "mathematically coherent", but I can imagine it used something flagrantly incoherent but really dead simple to implement. Like, for instance, the amount of some chemical or another coming in from the body, to indicate satiety, or to relax after physical exertion, or to indicate orgasm, or something like that.

Meanwhile, I went beyond that problem and outlined a solution, soon after I started working in this field in the mid-80s. And by 2006 I had clarified my ideas enough to present them at the AGIRI workshop held in Bethesda that year.

Link?

. It is a scandalously unjustified assumption, made very hard to attack by the fact that it is repeated so frequently that everyone believes it be true just because everyone else believes it.

I don't think that is an overstatement. If MIRI is basicatly wrong about UFs, then most of its case unravels. Why isnt the issue bring treated as a matter of urgency?

Since the utility function is approximated anyway, it becomes an abstract concept - especially in the case of evolved brains. For an evolved creature, the evolutionary utility function can be linked to long term reproductive fitness, and the value function can then be defined appropriately.

For a designed agent, it's a useful abstraction. We can conceptually rate all possible futures, and then roughly use that to define a value function that optimizes towards that goal.

It's really just a mathematical abstraction of the notion of X is better than Y. It's not worth arguing about. It's also proven in the real world - agents based on utility formalizations work. Well.

I am not sure why you say I am hung up on RL: you quoted that as the only mechanism to be discussed in the context, so I went with that.

Upon consideration, I changed my own usage of "Universal Reinforcement Learning Machine" to "Universal Learning Machine".

The several remaining uses of "reinforcement learning" are contained now to the context of the BG and the reward circuitry.

And you are (like many people) not correct to say that RL is the most general framework,

Again we are probably talking about very different RL conceptions. So to be clear, I summarized my general viewpoint of an ULM. I believe it is an extremely general model, that basically covers any kind of universal learning agent. The agent optimizes/steers the future according to some sort of utility function (which is extremely general), and self-optimization emerges naturally just by including the agent itself as part of the system to optimize.

Do you have a conception of a learning agent which does not fit into that framework?

or that there is good evidence for RL in the brain. That is a myth: the evidence is very poor indeed.

The evidence for RL in the brain - of the extremely general form I described - is indeed very strong, simply because any type of learning is just a special case of universal learning. Taboo 'reinforcement' if you want, and just replace with "utility driven learning".

AIXI specifically has a special reward channel, and perhaps you are thinking of that specific type of RL which is much more specific than universal learning. I should perhaps clarify and or remove the mention of AIXI.

A ULM - as I described - does not have a reward channel like AIXI. It just conceptually has a value and or utility function initially defined by some arbitrary function that conceptually takes the whole brain/model as input. In the case of the brain, the utility function is conceptual, in practice it is more directly encoded as a value function.