I reject the notion that one can factorize intelligence from goals, so that one could take a superintelligence and fuse it with a goal to optimize for paperclips.

I agree there aren't currently good arguments for why "one can factorize intelligence from goals", at least not in a strong sense, but what about Eliezer's thesis that value is complex and fragile, and therefore:

with most approaches to AGI (including neuromorphic or evolved, with very few exceptions such as WBE, which I wouldn't call AGI, just faster humans with more dangerous tools for creating an AGI), it's improbable that we end up with something close to human values, even if we try, and that greater optimization power of a design doesn't address this issue (while aggravating the consequences, potentially all the way to a fatal intelligence explosion)

I'm curious what you think of this line of argument.

Although I expect present-human culture to be preserved, as a matter of historical interest if not status quo, I'm not partisan enough to prioritize human values over the Darwinian imperative.

Does that mean you're familiar with Robin Hanson's "Malthusian upload" / "burning the cosmic commons" scenario but do not think it's a particularly bad outcome?

your time may be better spent trying to convince Ben Goertzel that there's a problem, since at least he's an immediate threat.

I'd guess that's been tried already, given that Ben was the Director of Research for SIAI (and technically Eliezer's boss) for a number of years.

...your time may be better spent trying to convince Ben Goertzel that there's a problem, since at least he's an immediate threat. ;)

I doubt it. I neither believe that people like Jürgen Schmidhuber are a risk, apart from a very abstract possibility.

The reason is that they are unable to show off some applicable progress on a par with IBM Watson or Siri. And in the case that they claim that their work relies on a single mathematical breakthrough, I doubt that it would be justified even in principle to be confident in that prediction.

In short, either their work is incrementally useful or is based on wild speculations about the possible discovery of unknown unknowns.

The real risks in my opinion are 1) that together they make many independent discoveries and someone builds something out of it 2) that a huge company like IBM, or a military project, builds something 3) the abstract possibility that some partly related field like neuroscience, or an unrelated field, provides the necessary insight to put two and two together.

I reject the notion that one can factorize intelligence from goals, so that one could take a superintelligence and fuse it with a goal to optimize for paperclips.

Do you mean that intelligence is fundamentally interwoven with complex goals?

...never completely turn its available resources into paperclips since that would mean no chance of more paperclips in the future;

Do you mean that there is no point at which exploitation is favored over exploration?

I'm not partisan enough to prioritize human values over the Darwinian imperative.

I am not sure what you mean, could you elaborate? Do you mean something along the lines of what Ben Goertzel says in the following quote:

But my gut reaction is: I’d choose humanity. As I type these words, the youngest of my three kids, my 13 year old daughter Scheherazade, is sitting a few feet away from me doing her geometry homework and listening to Scriabin Op. Fantasy 28 on her new MacBook Air that my parents got her for Hanukah. I’m not going to will her death to create a superhuman artilect. Gut feeling: I’d probably sacrifice myself to create a superhuman artilect, but not my kids…. I do have huge ambitions and interests going way beyond the human race – but I’m still a human.

You further wrote:

In summary, I'm just not worried about AI risk

What is your best guess at why people associated with SI are worried about AI risk?

I've heard many (probably most) AI-risk arguments, and failed to become worried...

If you would have to fix the arguments for the proponents of AI-risk, what would be the strongest argument in favor of it? Also, do you expect there to be anything that could possible change your mind about the topic and become worried?

I reject the notion that one can factorize intelligence from goals

Human intelligence has been successfully applied to achieve many goals which were not applicable or trainable in the environment of anscestral adaption such as designing, building, and operating cars, playing chess, sending people to the moon and back, and programming computers. It is clear the goals and intelligence can be factorized as a matter of simple observation.

(quite probably because I hold the cause of technological progress very dear to my heart and am thus heavily biased - at least I admit it!)

Upvoted for honesty.

I'd heard it said that the best human simulators required something like one laptop per neuron - I take it this is because cerebral cortex neurons are more demanding than any C. elegans neuron?

BTW, what's the paper about putting the zebrafish in the Matrix?