You know how there are people who, even though you could train them to carry out the steps of a Universal Turing Machine, you can't manage to teach them linear algebra, so there are problems they can't even represent compared to people who know linear algebra? I can't exhibit to you specifically what it is for obvious reasons, but there's going to be lots of stuff like that where a human brain just can't grok it, even though - like a sufficiently well-trained dog - we could be trained to carry out the operations of a UTM that did grok it, given infinite time and paper. You could train a chimp to simulate a human brain given infinite time and paper, I've little doubt. So what? There was still a huge jump in qualitative ability.

I just finished Diaspora, and realized that everything of Egan's I've read shares the fundamental assumption discussed here: that in the space of intelligence, there's "down" and there are infinitely many dimensions of "sideways" from where we are, but there's no meaningful "up".

But IMO Occam's Razor distinguishes strongly between the following two hypotheses:

1. There is a unique level of general intelligence, and it happens to be identical to the level required for hominids to build an agricultural civilization.

2. There is a bigger hierarchy of general intelligence, and hominids just got smarter until they crossed the "agricultural civilization" threshold.

Apart from theoretical commitments, the first seems to entail a contingent miracle that the second doesn't.

I think there are things that individual humans will never understand because we have finite lifespans and finite memory capacity. Anything that requires more storage space to understand than a human brain possesses will never be completely understood by a single unaided human. We might be able to build a system that includes at least one human that does understand it (in the style of Searle's Chinese Room); after all, a human with enough time, enough paper, and enough pencils can indeed execute any Turing machine, but I wouldn't want to claim that "I" understand everything that the program I'm running does.

Following both jimrandomh and MixedNuts, I think perhaps the disagreement could be expressed in terms of resource normalized learning curves.

A learning curve can be generated for a single domain and a single person, and the datasets that support them allow one to compare different people's ability to learn and the ease or difficulty of different domains. Here are good examples. I've actually seen these produced and compared in a startup and used to identify whose process should be mined for lessons to teach other analysts, and seeing the graphs was kind of surprising to me because I had expected them to be noisy and abstract, but using a moving average to smooth things out (or looking at day to day performance over several weeks) but they showed S curves as people exponentially got the hang of things with early insights allowing greater insights and then hit diminishing returns as the marginal utility of insights started to decrease. Also, there were obvious and visible between-person differences in the curves themselves.

Implicitly, all of those curves were assumed to have the same X-axis which was "human-brain-hours" or something like that. Human brains were assumed to be equivalent (at least they had the same startup burn rate per hour...) and so only the hours and days spent learning were being considered.

So imagine we work out a way to normalize processing power in terms of theoretically inter-convertable input to a physical computing process like transistors or joules or some such. You might be able to normalize learning curves between different systems using something relatively universal, like total energy expended on learning so far, or computing-element-gram-seconds or some such.

With such a measure you might be able to vividly see that a trillion joules spent learning a very hard task would produce vastly better results for any fixed resources you allocated to a "fundamentally smarter" process (assuming a task where the learning curve wouldn't flatten out too fast to make learning differences moot). At a certain point, especially if you're comparing processes with different starting points on the curve at "time=0", it might be pointless to allocate learning jobs to certain learning systems given time and resource constraints, and then we might say that the learning systems that weren't "worth" assigning a learning task were "categorically different"... but figuring out this categorical difference in practice would involve a lot of features of the context, like the opportunity costs implied by the number and quality of alternatives, the task you're considering, hard deadlines, and perhaps "throttling issues" where each system in the learning economy's performance per resource per time may become sub-linear in resources beyond a certain point. On throttling issues, Jacob Cannel has interesting thoughts based on the physics of computation.

This is a cool article and it made me think, so this is just a minor nitpick: I know "quantum leap" is commonly used to mean a large jump, but wouldn't that be a very tiny jump, if "quantum" is used in its usual (correct) sense?

(In a perfect world The Singularity Is Far would've been a Robin Hanson post...)

When people talk about cognitive hierarchies and UTMs and how humans are the first example of an entity on the smarter side of the general intelligence Rubicon, it's not obvious that someone like Eliezer would disagree. Or to speak more easily, I would not disagree, though with caveats that the metaphysics of this abstraction called 'computation' still seem confusing unto me. Unfortunately my weak impression is that the people who talk about such things seem to think these fringe Singularity people would disagree, because they hear pseudo-Vingean claims about unpredictability and the like. I'd imagine many of those pseudo-Vingean 'predictions' pattern match to that idea everyone has when they're eight years old about some kind of superconsciousness, the equivalent of universality on top of universality, a triumph over Goedel. They probably also pattern match the claims to even sillier ideas.

Again this is a guess, but it is non-negligibly likely that folk at least similar to Greg Egan have an implicit model of Singularitarian folk (whose diverse members and positions will unfortunately get lumped together to some rather non-negligibly biased extent) as having differing and very probably wrong intuitions about theoretical computer science, instead of having differing but at least plausible beliefs about rates/bursts of technological progress or problem solving ability as measured by biological subjective time. That said, if you asked Egan if he explicitly thinks that Eliezer explicitly thinks that super-Turingness-like-stuff is probable, I don't think he'd say yes, so it seems probable that my model is at least somewhat wrong.

I think that Aaronson's comments are correct, though the human thought serial speed-up thing is perhaps a misleading example for this post since the sort of singularity Eliezer et al are interested in is not really about serial speed-ups themselves so much as using speed-ups to search for more speed-ups among other things. Aaronson's P versus NP example is an okay theoretical example of an insight that smart Singularitarians might talk about as in-hindsight practically incompressible but still powerful enough to 'change the nature of the optimization game' as I think Eliezer put it.

That humans might in principle follow any argument a superintelligence could put forth up to the Goedelian limit is maybe true if you're willing to really really stretch the definition of human. This implies next to nothing about whether or not hardware-enhanced humans or even ems can stay ahead of de novo Bayesian seed AIs or even hacked-together but still non-biologically-originated AIs, whose architectures are wayyyyy more flexible and conducive to software improvements. I have a hard time coming up with a decent counterargument Greg Egan could make here conditional on his accepting the practical possibility of engineering AGI. I'm probably missing nuances of his position. I wish he could have a structured discussion with Eliezer. (Bloggingheads is too much like a debate, no one goes in expecting to change their mind.)

I agree that the chimpanzee-to-human phase transition example is potentially misleading unless there's something I'm missing. If you've seen it used in an SIAI paper in a way that doesn't mention the possible narrow-to-general intelligence phase transition argument, please link to it.