This is a disappointing post.

On atomism: the atomic theory of the pre-Socratics was just another in a series of essentially navel-gazing theories. Today we privilege and single out that theory for the obvious reasons, but it doesn't deserve much more praise than the alternatives. The philosophers involved had absolutely no way to test it and in fact weren't interested in doing so. And the alternatives were as convincing or more convincing, given the available evidence, as their speculation.

Suppose some SF writer in 1940s wrote a hack piece about an alien invasion where we turn out to live in "ten dimensions" and the aliens are coming out of the extra ones. And suppose string theory actually comes through and proves itself in another 20-30 years. Giving credit to the pre-Socratic atomists for figuring out atoms would be as silly as giving credit to that SF author for figuring out before everyone else that we really live in 10 dimensions.

On natural selection: the hard part with natural selection is not figuring out that it happens. It's figuring out that it happens and it's responsible for a large portion of evolutionary diversity. That's not obvious at all, to put it very mildly. In retrospect or not. And to even formulate that hypothesis - which, given only everyday knowledge, seems way too bizarre to even consider - requires a huge body of biological knowledge, taxonomy, anatomy, the Galapagos islands, etc. Darwin's achievement was not to say "hey, natural selection happens". It was to say "hey, I know that natural selection on the face of it looks like it could drive at most some small change within a species, but actually, THIS is the mechanism mostly responsible for EVERYTHING".

But more than that, you can't even talk about natural selection before you're convinced there's evolution. Biologists were by and large convinced there was evolution by the time Darwin showed up. How the hell would ancient Greeks be able to get there just by sitting down and thinking "from first principles"? You need to dissect hundreds of species and see how similar they are inside. You need fossil records and something to compare them to. You need geology to tell you that Earth is really old. You need Lamarck.

On intelligence explosion: not sure where to begin, maybe with "they could have realized that scientists would eventually come to understand these systems so long as scientific progress continued". They didn't have any conception of scientific progress. They didn't know anything about "scientists", either. And they had absolutely no basis for thinking that just because people are trying to understand e.g. intelligence, that they're actually going to succeed.

As pointed out by others, the atomism of Democritus is not a solid theory, but a blue vs. green position. One fellow thinks that if you divide something repeatedly, you'll never stop being able to divide it; another fellow disagrees with him, they come up with elaborate verbal justifications for their positions.

So why don't we consider some Greek theories that were actually solid theories? Aristarchos argued that, since the sun was so much bigger than the Earth, it made more sense for the Earth to orbit the sun than the sun to orbit the Earth.

Other mathematicians thought about this theory, and eventually responded with, "well, if that were true, we should expect to see parallax effects on other stars. We don't, so either everything orbits around the Earth or the stars are unimaginably far away."

Well, it turns out that the stars are unimaginably far away, and so the parallax effects are too weak to see with the naked eye. But the Greeks had their priors in the right place for the evidence they had access to.

Likewise, things like natural selection (as we understand it) and intelligence explosion don't seem like they're obvious without evidence to back them up, and could be reasonably dismissed without that evidence.

Intelligence explosion follows from physicalism and scientific progress and not much else.

You say it like it's a mathematical theorem or an experimentally tested physical model. In fact, there are plenty of caveats in the former approach, and there is a falsifying evidence in the latter (no recognizable super-intelligence on Earth or in space, billions of years after the Big Bang). Imagine that the Moore's law fizzles in the next few years, where will your intelligence explosion be? Or maybe there is a law that states that stupidity grows faster than intelligence, resulting in the world overrun by idiots, killing further progress.

I'm not sure it's quite accurate to say they didn't discover natural selection. They tended to believe a lot of other nonsense, though, which made it difficult to fully realize the idea.

Aristotle considered something like natural selection but rejected it because it didn't otherwise fit his worldview (which was opposed to atomism). Here he considers selection and variation, although he doesn't mention heritability explicitly.

Why then should it not be the same with the parts in nature, e.g. that our teeth should come up of necessity--the front teeth sharp, fitted for tearing, the molars broad and useful for grinding down the food--since they did not arise for this end, but it was merely a coincident result; and so with all other parts in which we suppose that there is purpose? Wherever then all the parts came about just what they would have been if they had come be for an end, such things survived, being organized spontaneously in a fitting way; whereas those which grew otherwise perished and continue to perish, as Empedocles says his 'man-faced ox-progeny' did.

Yet it is impossible that this should be the true view... action for an end is present in things which come to be and are by nature.

Source is http://classics.mit.edu/Aristotle/physics.2.ii.html, part 8. The reference to Empedocles is talking about http://en.wikipedia.org/wiki/Empedocles#Cosmogony, who arguably came closer, but it sounds like he had some pretty strange ideas about the origins of things. Anyway, I'd suggest that incorrect (scientific and metaphysical) beliefs were at least as problematic as lack of imagination in this case.

I don't know too much about Greek philosophy, though, so I'd be interested to hear someone explore this further.

http://intelligenceexplosion.com/ paints a pretty naive picture.

Designing artificial intelligences is a skill composed of many sub-skills of varying levels of difficulty.

Machines are better at many of those today.

To say "one day" creates the bizarre and totally incorrect idea that machine skill will overtake human skill on the specified task on one day - and after that things will go much faster. What is actually happening is that machine skill has been gradually overtaking human skill in a range of domains, one domain at a time. The process has been going on for many decades now. Composite tasks that involve many skills - like designing complex machines by mastering hardware and software engineering - will thus be accelerated gradually by the increase in performance of machines.

This appears to be a confusion propagated by those wanting to exaggerate the risks of machine intelligence. I have a web page about this and must have told people about this publicly a dozen times now, but few seem to listen, and consequently the same nonsense gets repeatedly pumped out, deluding wave after wave of newcomers about this point. Perhaps brevity excuses this instance, but it is beginning to look like a deliberate deception - to try and make things seem worse than they appear, by making the transition to intelligent machines seem more sudden than is likely.

This video shows the position that I am arguing against (at 18:43).

It might make good propaganda, but it is based on an inacurate picture of what it likely to happen. We know enough to see that already.

Did the ancient Greeks even have a concept of humans as varying over time? Certainly they didn't have a single machine which did anything even remotely like thinking.

I quite clearly remember growing up in a universe which was static. People even talked about mechanisms to create hyrdrogen in deep space in order to keep the density constant in light of the Hubble expansion. I REMEMBER seeing the big bang theory (not the TV show!) and just marveling at the audacity. How could the universe have started? What was here before?

And quite similarly I remember thinking there is "history" and there is "now." History had things like WWII and the bubonic plague in it, now had civil rights, the Soviet Union, and the Vietnam War. I was probably in my 30s when I finally realized that 1) we ARE in hisotry and 2) it ain't slowing down, if anything its speeding up.

When I put myself in the place of the greeks, humans would seem static, human culture would seem largely static, machines would seem darn near irrelevant to anything.

I can't imagine I could look forward to an intelligence explosion until I had some sense that looking backward there had been an intelligence explosion. Even now you can talk to Rabbis who will argue with you about evolution by asking you if you really believe you are descended from a monkey. Did ANYONE in ancient greece have any idea that we had any other relation to monkeys or horses or dogs other than sharing the planet with them?

Could democritus have predicted an Intelligence Explosion?

Only if he had been superhumanly able to follow arguments through to the maximum extent of their consequences.

The number and complexity of the intervening steps, listed by morendil, put “FOOM from Atoms alone" firmly beyond the predictive power from limited evidence exhibited by any historic person who in the end turned out to be right. Which is to say, beyond Democritus, as he perhaps marks the upper end of this scale. There is good reason to believe that there is not sufficient working memory available to members of species Homo Sapiens to discover “FOOM from Atoms alone.” I mean, none of the greatest scientists were near that good.

Intelligence explosion follows from physicalism and scientific progress and not much else.

I agree that physicalism affords the possibility of an Intelligence Explosion from the observed existence of minds, and the physical limits to the size and power of possible minds, compared to the size and power of observed minds.

• The laws of physics allow minds.
• The laws of physics put some upper limit on the size and power of minds, the limits are many orders of magnitude greater than those of observed minds.
• Therefore, there is plenty of room above us.

From this we can conclude that superintelligent minds are permitted by physics, but not necessarily that one can jump from intelligence to superintelligence through an intelligence explosion.

We know something about physical limits of Minds. Less is known about the limits to the progress of science, and it's more relevant proxy, engineering. We do not know if the maximally powerful engineering available to evolved minds is above the level required for initiating the chain-reaction of recursive self-improvement, initiating an intelligence explosion towards the physical limits for minds.

This begs the question: Is the ceiling of engineering prowess available to evolved minds lower than the floor of engineering prowess required for recursive self improvement?

To the best of my knowledge, this question is undecided.

Is there good evidence in one direction or the other?

Superintelligence by way of a controlled Intelligence Explosion seems harder than General Artificial Intelligence by Uploading by a large amount. Like intergalactic travel is harder than interstellar travel by a large amount.

An Intelligence Explosion may permitted under physicalism, yet be prohibited by cost and complexity, like intergalactic travel is permitted under physicalism, yet prohibited by cost and engineering complexity.

Thereby saving minds such as we, around the minimum level of intelligence required for general intelligence, from being displaced by minds that are powerful to the maximum degree that the limits-of-physics allow.

Safety wise initiating an Intelligence Explosion seems around as safe as going to the moon by riding on the heat and pressure of a nuclear chain reaction instead of on the tip of a large chemical rocket.

When I was a kid, my parents, especially my dad, used to talk about difficulties he had in raising me, and how they were related to difficulties that he had as a kid with his parents.

And I thought "that means I'll get to grow up better than my dad. And when I grow up and have kids, I'll have an advantage, and my kids will be able to grow up better off than me. And eventually my children's children's children will be rad as all get-out."

And while I hadn't really noticed until now, that was my point of reference for "intelligence explosion." It now seems clear to me that (a) most people haven't even begun to advance along this line, and/or see it as having some bounded destination, and (b) other types of changes will blow these out of the water, in the way that memetic evolution in humans blew genetic evolution out of the water. And honestly, that's scary. Because the improved-parenting intelligence improvement comes with a responsibility and morality improvement as well, built right in. Because it's a people-improvement, rather than mind-improvement.

But this type of intelligence increase, not really an explosion, seems pretty Greek-accessible to me, in the same way that "Science must pass Judaism" was accessible to Eliezer as a kid.

Since materialists had to believe that human intelligence resulted from the operation of mechanical systems located in the human body . . .

What little I know about ancient Greek thought makes me doubt that they would have been equipped to follow up on the full implications of "materialism" in the modern sense. Following Newton, we're now all completely accustomed to the idea that everything on Earth is governed by the same basic principles as everything else in the universe. We're used to the idea of matter and energy and forces being quantified into precise algebraic formulas, and even random events can be described with some precision. But the Greeks, for all their knowledge of geometry, did not know algebra or probability.

As I understand it, the prevailing view in those days was that the things above the moon -- the stars and planets -- followed mathematical patterns (moving in circles, of course, because circles are so obviously terrific). However, on Earth, in our imperfect sublunary sphere, things had tendencies and impulses. Fire more or less wanted to go up, and water down, more or less in the same way that a squirrel wants to run away from a dog.

According to Wikipedia :

Aristotle also argues that the mind (only the agent intellect) is immaterial, able to exist without the body, and immortal. ...One argument for its immaterial existence runs like this: if the mind were material, then it would have to possess a corresponding thinking-organ. And since all the senses have their corresponding sense-organs, thinking would then be like sensing. But sensing can never be false, and therefore thinking could never be false. And this is of course untrue. Therefore, Aristotle concludes, the mind is immaterial.

Nitpick about the image at facingthesingularity.com, just above the title : Great image if you don't know the story, but can convey quite the wrong ideas to those who played the game it refers to (Mass Effect). (Spoiler ahead.) The space city this image depicts is a the centre of a galactic trap given by malevolent gods. This trap presents itself as a galactic transport network, which helps the development of a galactic civilization. (By the way, nobody understand how it works.) Then it "activates" and help the bad guys wipe out this civilization. Then they wait for the next, rinse and repeat. (Don't ask what is the need of such a complicated plot for such powerful beings.)

Hopefully, those who notice this will also notice that you totally didn't mean that.

Atomism had been "predicted" by Democritus, and still, it wasn't accepted until Dalton. This is a big piece of evidence that atomism wasn't obvious for people unfamiliar with weighing chemical compounds. The same would apply to natural selection. Saying that it is obvious once you understand it smells of hindsight bias; ideas often look obvious when known to be true. The ancient Greeks didn't know that learned traits are not inherited, nor they had much evidence for this fact. They didn't know that the boundaries between species aren't unsurmountable, nor they had much evidence: almost no knowledge of fossils, not particularly good understanding of breeding and its history. It was perfectly reasonable for an ancient Greek philosopher to assume that a population of goats will forever remain a population of goats. The individual goats may vary, but the variability could be always constrained by some Platonic prototype of goat category whose boundaries are defined and maintained by gods or other eternal forces of nature. Not to mention that the important part of natural selection idea - that it is the driving force behind evolution - doesn't automatically follow from only heritability, variation and selection. In a stable environment the populations may well occupy local minima in the fitness landscape and remain stable for a long time.

But what I am most bothered about is that this post appears as an attempt to declare natural selection and intelligence explosion as equally obvious, from today's perspective.

Why didn't they discover natural selection, too?

This is precisely worded to not address the issue of whether or not they could have ruled out some amount of Lamarkian inheritance, yes?

Okay. Now we know, intelligence explosion will most likely happen. Also we know some other important things in this context.

What is still unheard of, not published widely - and can be now predicted from this high point of view?