I’m going to quote things I agreed with or things that persuaded me or that worried me.

Okay, to start off, when I first read about this in Intelligence Explosion: Evidence and Import, Facing the Intelligence Explosion, Intelligence Explosion and Machine Ethics it just felt like self-evident and I’m not sure how thoroughly I went through the presuppositions during that time so Kruel could have very easily persuaded me about this. I don’t know much about the technical process of writing an AGI so excuse me if I get something wrong about that particular thing.

Are the conclusions justified? Are the arguments based on firm ground? Would their arguments withstand a critical inspection or examination by a third party, peer review? Are their estimations reflectively stable? How large is the model uncertainty?

Most of their arguments are based on a few conjectures and presuppositions about the behavior, drives and motivations of intelligent machines and the use of probability and utility calculations to legitimate action.

It’s founded on many, many assumptions not supported by empirical data, and if even one of them was wrong the whole thing collapses down. And you can’t really even know how many unfounded sub-assumptions there are in these original assumptions. But when I started thinking about it could be that it’s impossible to reason about those kind of assumptions if you do it any other way than how MIRI currently does it. Needing to formalize a mathematical expression before you can do anything like Kruel suggested is a bit unfair.

The concept of an intelligence explosion, which is itself a logical implication, should not be used to make further inferences and estimations without additional evidence.

The likelihood of a gradual and controllable development versus the likelihood of an intelligence explosion.

The likelihood of unfriendly AI versus friendly AI as the outcome of practical AI research.

I don’t see why the first AIs resembling general intelligences would be very powerful so practical AGI research is probably somewhat safe in the early stages.

The ability of superhuman intelligence and cognitive flexibility as characteristics alone to constitute a serious risk given the absence of enabling technologies like advanced nanotechnology.

That some highly intelligent people who are aware of the Singularity Institute’s position do not accept it.

How is an AI going to become a master of dark arts and social engineering in order to persuade and deceive humans?

How is an AI going to coordinate a large scale conspiracy or deception, given its initial resources, without making any suspicious mistakes along the way?

Are those computational resources that can be hacked applicable to improve the general intelligence of an AI?

Does throwing more computational resources at important problems, like building new and better computational substrates, allow an AI to come up with better architectures so much faster as to outweigh the expenditure of obtaining those resources, without hitting diminishing returns?

This I would like to know, how scalable is intelligence?

How does an AI brute-force the discovery of unknown unknowns?

(I thought maybe by dedicating lots of computation to a very large numbers of random scenarios)

How is an AI going to solve important problems without real-world experimentation and slow environmental feedback?

(maybe by simulating the real world environment)

How is an AI going to build new computational substrates and obtain control of those resources without making use of existing infrastructure?

How is an AI going to cloak its actions, i.e. its energy consumption etc.?

The existence of a robot that could navigate autonomously in a real-world environment and survive real-world threats and attacks with approximately the skill of C. elegans. A machine that can quickly learn to play Go on its own, unassisted by humans, and beat the best human players.

A theorem that there likely exists a information theoretically simple, physically and economically realizable, algorithm that can be improved to self-improve explosively. Prove that there likely are no strongly diminishing intelligence returns for additional compute power.

Show how something like expected utility maximization would actually work out in practice.

Conclusive evidence that current research will actually lead to the creation of superhuman AI designs equipped with the relevant drives that are necessary to disregard any explicit or implicit spatio-temporal scope boundaries and resource limits.

http://kruel.co/2013/01/04/should-you-trust-the-singularity-institute/

Thoughts on this article. I read about the Nurture Assumption in Slate Star Codex and it probably changed my priors on this. If it really is true and one dedicated psychologist could do all that, then MIRI probably could also work because artificial intelligence is such a messy subject that a brute force approach using thousands of researchers in one project probably isn't optimal. So I probably wouldn’t let MIRI code an AGI on its own (maybe) but it could give some useful insight that other organizations are not capable of.

But I have to say that I’m more favorable to the idea now than when I made that post. There could be something in the idea of intelligence explosion, but there are probably several thresholds in computing power and in the practical use of the intelligence. Like Squark said above, the research is still interesting and if continued will probably be useful in many ways.

love,

the father of the unmatchable (ignore this, I'm just trying to build a constructive identity)