Recent published SI work concerns AI safety. They have not recently published results on AGI, to whatever extent that is separable from safety research, for which I am very grateful. Common optimization algorithms do apply to mathematical models, but that doesn't limit their real world use; an implemented optimization algorithm designed to work with a given model can do nifty things if that model roughly captures the structure of a problem domain. Or to put it simply, models model things. SI is openly concerned with exactly that type of optimization, and how it becomes unsafe if enough zealous undergrads with good intentions throw this, that, and their grandmother's hippocampus into a pot until it supposedly does fantastic venture capital attracting things. The fact that SI is not writing papers on efficient adaptive particle swarms is good and normal for an organization with their mission statement. Foom was a metaphorical onomatopoeia for an intelligence explosion, which is indeed a commonly used sense of the term "technological singularity".

The highly specific predictions should be lowered in their probability when updating on the statement like 'unpredictable'.

That depends what your initial probability is and why. If it already low due to updates on predictions about the system, then updating on "unpredictable" will increase the probability by lowering the strength of those predictions. Since destruction of humanity is rather important, even if the existential AI risk scenario is of low probability it matters exactly how low.

This of course has the same shape as Pascal's mugging, but I do not believe that SI claims are of low enough probability to be dismissed as effectively zero.

Not everything is equally easy to describe as equations.

That was in fact my point, which might indicate that we are likely to be talking past each other. What I tried to say is that an artificial intelligence system is not necessarily constructed as an explicit optimization process over an explicit model. If the model and the process are implicit in its cognitive architecture then making predictions about what the system will do in terms of a search are of limited usefulness.

And even talking about models, getting back to this:

cutting down the solution space and cutting down the model

On further thought, this is not even necessarily true. The solution space and the model will have to be pre-cut by someone (presumably human engineers) who doesn't know where the solution actually is. A self-improving system will have to expand both if the solution is outside them in order to find it. A system that can reach a solution even when initially over-constrained is more useful than the one that can't, and so someone will build it.

I think you have a very narrow vision of 'unstable'.

I do not understand what you are saying here. If you mean that by unstable I mean a highly specific trajectory a system that lost stability will follow, then it is because all those trajectories where the system crashes and burns are unimportant. If you have a trillion optimization systems on a planet running at the same time you have to be really sure that nothing can't go wrong.

I just realized I derailed the discussion. The whole AGI in specialized AI world is irrelevant to what started this thread. In the sense of chronology of being developed I cannot tell how likely it is that AGI could overtake specialized intelligences. It really depends whether there is a critical insight missing for the constructions of AI. If it is just an extension of current software then specialized intelligences will win for reasons you state. Although some of the caveats I wrote above still apply.

If there is a critical difference in architecture between current software and AI then whoever hits that insight will likely overtake everyone else. If they happen to be working on AGI or even any system entangled with the real world, I don't see how once can guarantee that the consequences will not be catastrophic.

Too much anthropomorphization.

Well, I in turn believe you are applying overzealous anti-anthropomorphization. Which is normally a perfectly good heuristic when dealing with software, but the fact is human intelligence is the only thing in "intelligence" reference class we have, and although AI will almost certainly be different they will not necessarily be different in every possible way. Especially considering the possibility of AI that are either directly base on human-like architecture or even are designed to directly interact with humans, which requires having at least some human-compatible models and behaviours.

Hm, most of the immediate strategies SI is considering going forward strike me as fairly general:

http://lesswrong.com/r/discussion/lw/cs6/how_to_purchase_ai_risk_reduction/

They're also putting up these strategies for public scrutiny, suggesting they're open to changing their plans.

If you're referring to sponsoring an internal FAI team, Luke wrote:

I don't take it to be obvious that an SI-hosted FAI team is the correct path toward the endgame of humanity "winning." That is a matter for much strategic research and debate.

BTW, I wish to reinforce you for the behavior of sharing a dissenting view (rationale: view sharing should be agnostic to dissent/assent profile, but sharing a dissenting view intuitively risks negative social consequences, an effect that would be nice to neutralize), so I voted you up.

A suggestion: it may be a bad idea to use word 'artificial intelligence' in the name without qualifiers, as to serious people in the field

• the 'artificial intelligence' has much, much broader meaning than what SI is concerning itself with

• there is very significant disdain for the commonplace/'science fiction' use of 'artificial intelligence'

• Center for Helpful Artificial Optimizer Safety (CHAOS)
• Center for Slightly Less Probable Extinction
• Freindly Optimisation Of the Multiverse (FOOM)
• Yudkowsky's Army
• The Center for World Domination
• Pinky and The Brain Institute
• Cyberdyne Systems

I have direct experience of someone highly intelligent, a prestigious academic type, dismissing SI out of hand because of its name. I would support changing the name.

Almost all the suggestions so far attempt to reflect the idea of safety or friendliness into the name. I think this might be a mistake, because for people who haven't thought about it much, this invokes images of Hollywood. Instead, I propose having the name imply that SI does some kind of advanced, technical research involving AI and is prestigious, perhaps affiliated with a university (think IAS).

Center for Advanced AI Research (CAAIR)

From what I gathered SI's relevance rests upon an enormous conjunction of implied and a very narrow approach as solution, both of which were decided upon significant time in the past. Subsequently, truly microscopic probability of relevance is easily attained; I estimate at most 10^-20 due to multiple use of narrow guesses into a huge space of possibilities.

Just because it doesn't do exactly what you want doesn't mean it is going to fail in some utterly spectacular way.

I certainly agree, and I am not even sure what the official SI position is on the probability of such failure. I know that Eliezer in hist writing does give the impression that any mistake will mean certain doom, which I believe to be an exaggeration. But failure of this kind is fundamentally unpredictable, and if a low probability even kills you, you are still dead, and I think that it is high enough that the Friendly AI type effort would not be wasted.

(ultimately, for solutions to systems of equations)

That is true in the trivial sense that everything can be described as equations, but when thinking how computation process actually happens this becomes almost meaningless. If the system is not constructed as a search problem over high dimensional spaces, then in particular its failure modes cannot be usefully thought about in such terms, even if it is fundamentally isomorphic to such a search.

that'll be a piece of software created with very well defined model of changes to itself

Or it will be created by intuitively assembling random components and seeing what happens. In which case there is no guarantee what it will actually do to its own model or even to what it is actually solving for. Convincing AI researches to only allow an AI to self modify when it is stable under self modification is a significant part of the Friendly AI effort.

Everyone wants artificial rainman.

There are very few statements that are true about "everyone" and I am very confident that this is not one of them. Even if most people with actual means to build one want specialized and/or tool AIs, you only need one unfriendly-successful AGI project to potentially cause a lot of damage. This is especially true as both hardware costs fall and more AI knowledge is developed and published, lowering the entry costs.

I don't see why expect general intelligence to suddenly overtake specialized intelligences;

To be dangerous AGI doesn't have to overtake specialized intelligences, it has to overtake humans. Existence of specialized AIs is either irrelevant or increases the risks from AGI, since they would be available to both, and presumably AGIs would have lower interfacing costs.

There's probably a lot of low hanging fruit, for example use of correct priors, e.g. given Gaussian prior distribution, a quite strong proof should be needed before you should believe someone (including yourself) has very high intelligence or expertise on a task.

Furthermore, many ways of evaluating people are to some extent self reinforcing as the people being evaluated are aware of evaluation. A smart person or expert can relatively cheaply demonstrate intelligence and/or expertise, in some way that provides very strong evidence, and will do so even for relatively moderate payoffs. Likewise, very selfless people can cheaply (in terms of their utilons) help strangers, and will do so for lower payoffs than people with more selfish utilities.

Other issue is gradual updates on huge amounts of weak, possibly non-independent evidence.

I think what may be confusing about expected outcome is the name. You don't actually expect to get 5 dollars out of this :) . You don't even expect to get, say, 5 million dollars after 1 million games, such would be rather unlikely. You do expect to get 5$ per game if you played infinitely many games, though, and if you are playing such games on small amounts of money you can choose the game to play based on expected outcome.