Following both jimrandomh and MixedNuts, I think perhaps the disagreement could be expressed in terms of resource normalized learning curves.
A learning curve can be generated for a single domain and a single person, and the datasets that support them allow one to compare different people's ability to learn and the ease or difficulty of different domains. Here are good examples. I've actually seen these produced and compared in a startup and used to identify whose process should be mined for lessons to teach other analysts, and seeing the graphs was kind of surprising to me because I had expected them to be noisy and abstract, but using a moving average to smooth things out (or looking at day to day performance over several weeks) but they showed S curves as people exponentially got the hang of things with early insights allowing greater insights and then hit diminishing returns as the marginal utility of insights started to decrease. Also, there were obvious and visible between-person differences in the curves themselves.
Implicitly, all of those curves were assumed to have the same X-axis which was "human-brain-hours" or something like that. Human brains were assumed to be equivalent (at least they had the same startup burn rate per hour...) and so only the hours and days spent learning were being considered.
So imagine we work out a way to normalize processing power in terms of theoretically inter-convertable input to a physical computing process like transistors or joules or some such. You might be able to normalize learning curves between different systems using something relatively universal, like total energy expended on learning so far, or computing-element-gram-seconds or some such.
With such a measure you might be able to vividly see that a trillion joules spent learning a very hard task would produce vastly better results for any fixed resources you allocated to a "fundamentally smarter" process (assuming a task where the learning curve wouldn't flatten out too fast to make learning differences moot). At a certain point, especially if you're comparing processes with different starting points on the curve at "time=0", it might be pointless to allocate learning jobs to certain learning systems given time and resource constraints, and then we might say that the learning systems that weren't "worth" assigning a learning task were "categorically different"... but figuring out this categorical difference in practice would involve a lot of features of the context, like the opportunity costs implied by the number and quality of alternatives, the task you're considering, hard deadlines, and perhaps "throttling issues" where each system in the learning economy's performance per resource per time may become sub-linear in resources beyond a certain point. On throttling issues, Jacob Cannel has interesting thoughts based on the physics of computation.
In this post I question one disagreement between Eliezer Yudkowsky and science fiction author Greg Egan.
In his post Complex Novelty, Eliezer Yudkowsky wrote in 2008:
An interview with Greg Egan in 2009 confirmed this to be true:
The theoretical computer scientist Scott Aaronson wrote in a post titled 'The Singularity Is Far':
An argument that is often mentioned is the relatively small difference between chimpanzees and humans. But that huge effect, increase in intelligence, rather seems like an outlier and not the rule. Take for example the evolution of echolocation, it seems to have been a gradual progress with no obvious quantum leaps. The same can be said about eyes and other features exhibited by biological agents that are an effect of natural evolution.
Is it reasonable to assume that such quantum leaps are the rule, based on a single case study? Are there other animals that are vastly more intelligent than their immediate predecessors?
What reason do we have to believe that a level above that of a standard human, that is as incomprehensible to us as higher mathematics is to chimps, does exist at all? And even if such a level is possible, what reason do we have to believe that artificial general intelligence could consistently uplift itself to a level that is incomprehensible to its given level?
To be clear, I do not doubt the possibility of superhuman AI or EM's. I do not doubt the importance of "friendliness"-research and that it will have to be solved before we invent (discover?) superhuman AI. But I lack the expertise to conclude that there are levels of comprehension that are not even fathomable in principle.
In Complexity and Intelligence, Eliezer wrote:
If we were able to specify the laws of physics and one of the effects of their computation would turn out to be superhuman intelligence that is incomprehensible to us, what would be the definition of 'incomprehensible' in this context?
I can imagine quite a few possibilities of how a normal human being can fail to comprehend the workings of another being. One example can be found in the previously mentioned article by Scott Aaronson:
Mr. Aaronson also provides another fascinating example in an unrelated post ('The T vs. HT (Truth vs. Higher Truth) problem'):
Those two examples provide evidence for the possibility that even beings who are fundamentally on the same level might yet fail to comprehend each other.
An agent might simply be more knowledgeable or lack certain key insights. Conceptual revolutions are intellectually and technologically enabling to the extent that they seemingly spawn quantum leaps in the ability to comprehend certain problems.
Faster access to more information, the upbringing, education, or cultural and environmental differences and dumb luck might also intellectually remove agents with similar potentials from each other to an extent that they appear to reside on different levels. But even the smartest humans are dwarfs standing on the shoulders of giants. Sometimes the time is simply ripe, thanks to the previous discoveries of unknown unknowns.
As mentioned by Scott Aaronson, the ability to think faster, but also the possibility to think deeper by storing more data in one's memory, might cause the appearance of superhuman intelligence and incomprehensible insight.
Yet all of the above merely hints at the possibility that human intelligence can be amplified and that we can become more knowledgeable. But with enough time, standard humans could accomplish the same.
What would it mean for an intelligence to be genuinely incomprehensible? Where do Eliezere Yudkowsky and Greg Egan disagree?