Deep Blue just brute forces the game tree (more-or-less). Obviously, this is not at all how humans play chess. Deep Blue's evaluation for a specific position is more "intelligent", but it's just hard-coded by the programmers. Deep Blue didn't think of it.
I can't remember right off hand, but there's some AI researcher (maybe Marvin Minsky?) who pointed out that people use the word "intelligence" to describe whatever humans can do for which the underlying algorithms are not understood. So as we discover more and more algorithms for doing intelligent stuff, the goalposts for what constitutes "intelligence" keep getting moved. I think I remember one particular prominent intellectual who, decades ago, essentially declared that when chess could be played better by a computer than a human, the problem of AI would be solved. Why was this intellectual surprised? Because he didn't realize that there were discoverable, implementable algorithms that could be used to complete the action of playing chess. And in the same way, there exist algorithms for doing all the other thinking that people do (including inventing algorithms)... we just haven't discovered and refined them the way we've discovered and refined chess-playing algorithms.
(Maybe you're one of those Cartesian dualists who thinks humans have souls that don't exist in physical reality and that's how they do their thinking? Or you hold some other variation of the "brains are magic" position? Speaking of magic, that's how ancient people thought about lightning and other phenomena that are well-understood today... given that human brains are probably the most complicated natural thing we know about, it's not surprising that they'd be one of the last natural things for us to understand.)
The output of a machine-learning algorithm is basically a black box.
Hm, that doesn't sound like an accurate description of all machine learning techniques. Would you consider the output of a regression a black box? I don't think I would. What's your machine learning background like, by the way?
Anyway, even if it's a black box, I'd say it constitutes appreciable progress. It seems like you are counting it as a point against chess programs that we know exactly how they work, and a point against Watson that we don't know exactly how it works.
There are impressive results which look like intelligence, which are improving incrementally over time. There is no progress towards an efficient "intelligence algorithm", or "understanding how intelligence works".
My impression is that many, if not most, experts in AI see human intelligence as essentially algorithmic and see the field of AI as making slow progress towards something like human intelligence (e.g. see this interview series). Are you an expert in AI? If not, you are talking with an awful lot of certainty for a layman.
I think I remember one particular prominent intellectual who, decades ago, essentially declared that when chess could be played better by a computer than a human, the problem of AI would be solved.
Hofstadter, in Godel, Escher, Bach?
Maybe you're one of those Cartesian dualists who thinks humans have souls that don't exist in physical reality and that's how they do their thinking
Not at all. Brains are complicated, not magic. But complicated is bad enough.
Would you consider the output of a regression a black box?
In the sense that we don't understa...
Claim: The first human-level AIs are not likely to undergo an intelligence explosion.
1) Brains have a ton of computational power: ~86 billion neurons and trillions of connections between them. Unless there's a "shortcut" to intelligence, we won't be able to efficiently simulate a brain for a long time. http://io9.com/this-computer-took-40-minutes-to-simulate-one-second-of-1043288954 describes one of the largest computers in the world simulating 1s of brain activity in 40m (i.e. this "AI" would think 2400 times slower than you or me). The first AIs are not likely to be fast thinkers.
2) Being able to read your own source code does not mean you can self-modify. You know that you're made of DNA. You can even get your own "source code" for a few thousand dollars. No humans have successfully self-modified into an intelligence explosion; the idea seems laughable.
3) Self-improvement is not like compound interest: if an AI comes up with an idea to modify it's source code to make it smarter, that doesn't automatically mean it will have a new idea tomorrow. In fact, as it picks off low-hanging fruit, new ideas will probably be harder and harder to think of. There's no guarantee that "how smart the AI is" will keep up with "how hard it is to think of ways to make the AI smarter"; to me, it seems very unlikely.