I think XiXiDu's arguments are an over-reach but I think they contain an element of truth that is functionally important and that you're not giving him credit for. The place in your response where the insight is lurking seems to me like this bit:
The same is true of "General Lee won his battles because he was intelligent". It distinguishes the case in which he was a tactical genius from the case where he just had overwhelming numbers, or was very lucky, or had superior technology.
I understand what I'd expect to see if a victory came from overwhelming numbers or superior technology. Overwhelming numbers would be easy to quantify and plug into a very coarse grained victory predicting battled simulation. Superior technology would be trickier, but could probably be accounted for in terms of higher rate of fire, faster supply lines, better communication, and more detailed simulation of these and similar technological elements.
However, I'd be hard pressed to quantify "tactical ability" or "luckiness" other than by pitting generals against each other over and over and seeing who won, kinda the way strength in the games of go and chess are assessed, except in the one case attributing the outcomes to brains and in the other to noise. Perhaps I could simulate everything really precisely, including their minds and give the one with more "tactical ability" more clock cycles and RAM? But it would be tricky. Some real world algorithms actually do better with fewer clock cycles, because they over fit the training data if you give them enough rope to hang themselves with.
Fiction generally communicates more vividly when the elements of the story shown to the reader rather than told to the reader. "His brow furrowed and his eyes flicked back and forth." vs "He wasn't good at hiding it when the surprise made him suspicious." Telling communicates a theory about what happened without bothering with the evidence; showing gives you the evidence. When writers of fiction try to show superintelligence, they run into a problem that can be solved in various ways, but to some degree it amounts to operationalizations of the hypothesis that a particular agent in the story is really smart. I've been collecting fiction and "non-fiction about the fiction" in this area for a while and one of them seems appropriate here.
Charlie Stross has written much about superintelligence, so he's spent a fair amount of time trying to figure out what you'd expect to observe, and one of his theories on this score has been floating around for a while:
On a related note, I once heard Stross talk about how to write superintelligences, and he gave an illustrative example, paraphrased: "When I need to take my cat to the vet, I bring out the cat carrier. The cat knows what this means, and then runs for the cat door. He is then very surprised to discover that it's closed and locked. To him, these two events are totally coincidental. I think that it's easy to write a story about a superintelligence: just have any humans that try to act against it constantly surprised by apparent coincidences that turn out to all have been the superintelligence's fault in the end."
Suppose I'm trying to thwart a superintelligence, as an exercise, and I end up tripping on a rock at the key moment. Afterwards, my expectations is that the superintelligence hasn't, for example, saturated my environment with rocks with hidden mechanisms that can shift on command, causing me to trip when it's observed to be required because that would be wasteful and dumb. Instead I expect it to have placed a handful of such rocks in just the right places, and to be able to explain the basis for placing the rocks in those particular locations, and I expect its explanation to make sense in retrospect. It might take me three months to read and ponder and verify the explanation, but then at the end I should say "yeah, that was all just common sense and keen observation, iterated over and over".
What I expect, in other words, is that there exists predictable structure and pattern in the world that I'm just not noticing right now, but that I hypothesize someone else could notice if they were integrating more information, using better modeling, at faster clock cycles, in a more goal directed fashion. The hypothesis of "effective superintelligence" thus pre-supposes that there's this "low hanging idea fruit" out of our human reach for really banal reasons like having 7 plus or minus 2 working memory registers (rather than 15), or living 70 years rather than 140 (and thus having to curtail learning and go to seed equivalently earlier in our life cycle).
Another way to explain the issue in more vivid terms, closer to near mode... Imagine that some general wins four seemingly even battles in a row and gives a lot of post hoc reasoning about how and why he won. "Yes," he concludes while being quite tall and quite male and buffing his nails against his chest, "I guess I'm just a genius. Maybe you should make me president given how awesome I am." But one in sixteen equivalently competent generals would appear the same. And post hoc explanations are disturbingly easy to accept. Maybe the general just got lucky? (The luck hypothesis being Fermi's side in his famous conversation with Groves.) Or maybe he actually is particularly skilled but in a domain limited way? How would you tell whether it was luck, or not? How would you tell if it was domain agnostic, or not? Especially, how would you tell without having domain expertise yourself?
Imagine Omega created a duplicate earth exactly like ours (geography, ecology, etc) except humans were genetically adjusted so they had 15 working memory registers and lived to 140, and they attended school till they were 45 as a casual matter so they had to-us-encyclopedic knowledge of "the universe in general and how it works". Suppose one of them is transported here at age 50 (looking like a 25 year old). After they got over the squalor and amorality and the fact that we all resented them for being better than us, they might take over the world. Sure... that could be. Or instead they might "flirt with leftist causes and be briefly in the news after being arrested for involvement in a socialist rally that turned into a riot. After that, nothing."*.
Can you imagine a super intelligence whose long term result is "after that, nothing"? If value of information calculations exhibit diminishing marginal utility in the general case, this is precisely what I would expect. The early swift and meaningful rise happens when truly valuable information is being acquired faster than normal, but once the prodigy had the basic gist of the world in their head they would be pretty similar to everyone else with the same core knowledge. And yet if I saw someone getting better than normal outcomes in some domain that I didn't understand very well "superior intelligence" would be a tempting hypothesis. "Luck" or "intelligence"? How do you tell the difference from a distance? I don't have an answer but I think that generic falsification criteria to answer this question in advance in a domain agnostic way would be very valuable.
How would you tell whether it was luck, or not? How would you tell if it was domain agnostic, or not? Especially, how would you tell without having domain expertise yourself?
Given that all knowledge is probabilistic, it seems to me that I should believe there's a 93% chance he's skilled and a 7% chance he's lucky assuming equal prior probability. You could probably up your certainty a little by investigating whether other generals thought his tactics were brilliant or stupid, whether he has related skills like being good at chess and wargames, what his ...
The failures of phlogiston and vitalism are historical hindsight. Dare I step out on a limb, and name some current theory which I deem analogously flawed?
I name artificial intelligence or thinking machines - usually defined as the study of systems whose high-level behaviors arise from "thinking" or the interaction of many low-level elements. (R. J. Sternberg quoted in a paper by Shane Legg: “Viewed narrowly, there seem to be almost as many definitions of intelligence as there were experts asked to define it.”) Taken literally, that allows for infinitely many degrees of intelligence to fit every phenomenon in our universe above the level of individual quarks, which is part of the problem. Imagine pointing to a chess computer and saying "It's not a stone!" Does that feel like an explanation? No? Then neither should saying "It's a thinking machine!"
It's the noun "intelligence" that I protest, rather than to "evoke a dynamic state sequence from a machine by computing an algorithm". There's nothing wrong with saying "X computes algorithm Y", where Y is some specific, detailed flowchart that represents an algorithm or process. "Thinking about" is another legitimate phrase that means exactly the same thing: The machine is thinking about a problem, according to an specific algorithm. The machine is thinking about how to put elements of a list in a certain order, according to the a specific algorithm called quicksort.
Now suppose I should say that a problem is explained by "thinking" or that the order of elements in a list is the result of a "thinking machine", and claim that as my explanation.
The phrase "evoke a dynamic state sequence from a machine by computing an algorithm" is acceptable, just like "thinking about" or "is caused by" are acceptable, if the phrase precedes some specification to be judged on its own merits.
However, this is not the way "intelligence" is commonly used. "Intelligence" is commonly used as an explanation in its own right.
I have lost track of how many times I have heard people say, "an artificial general intelligence would have a genuine intelligence advantage" as if that explained its advantage. This usage fits all the checklist items for a mysterious answer to a mysterious question. What do you know, after you have said that its "advantage" is "intelligence"? You can make no new predictions. You do not know anything about the behavior of real-world artificial general intelligence that you did not know before. It feels like you believe a new fact, but you don't anticipate any different outcomes. Your curiosity feels sated, but it has not been fed. The hypothesis has no moving parts - there's no detailed internal model to manipulate. Those who proffer the hypothesis of "intelligence" confess their ignorance of the internals, and take pride in it; they contrast the science of "artificial general intelligence" to other sciences merely mundane.
And even after the answer of "How? Intelligence!" is given, the practical realization is still a mystery and possesses the same sacred impenetrability it had at the start.
A fun exercise is to eliminate the explanation "intelligence" from any sentence in which it appears, and see if the sentence says anything different:
Another fun exercise is to replace "intelligence" with "magic", the explanation that people had to use before the idea of an intelligence explosion was invented:
Does not each statement convey exactly the same amount of knowledge about the phenomenon's behavior? Does not each hypothesis fit exactly the same set of outcomes?
"Intelligence" has become very popular, just as saying "magic" used to be very popular. "Intelligence" has the same deep appeal to human psychology, for the same reason. "Intelligence" is such a wonderfully easy explanation, and it feels good to say it; it gives you a sacred mystery to worship. Intelligence is popular because it is the junk food of curiosity. You can explain anything using intelligence , and so people do just that; for it feels so wonderful to explain things. Humans are still humans, even if they've taken a few science classes in college. Once they find a way to escape the shackles of settled science, they get up to the same shenanigans as their ancestors, dressed up in the literary genre of "science" but still the same species psychology.