Will Pearson: are you suggesting that the simplest algorithm for intelligence is too large to fit in human memory?
Dude. Dude. No wonder you've been so emphatic in your denunciations of mysterious answers to mysterious questions.
Regarding the first reply here (a year later...): perhaps there is another problem visible here, the problem of when advice is too plain. The story advises in a fashion so transparently evident that even SHRDLU could get it: the poor student quite literally wasn't looking at anything, so Pirsig/Phædrus gave her a topic so mundane that she had to go down and see for herself. If Zen and the Art of Motorcycle Maintenance were a math textbook, the rule would be clear: "if you examine something, you will have something to say about it." But because writing is a mysterious art, it is assumed that the moral of a story about writing must be mysterious as well.
(Oddly, I never fell prey to this with the visual arts. I thank whoever told me about the negative-space/outline trick - that worked so well that I cached "drawing is seeing" instead.)
You're right that he should be able to engage standard critiques, Zubon, but if my (negligible) experience with the philosophy of free will is any indication, many "standard critiques" are merely exercises in wooly thinking. It's reasonable for him to step back and say, "I don't have time to deal with this sort of thing."
Wow, there are a lot of nihilists here.
I answered on my own blog, but I guess I'm sort of with dloye at 08:54: I'd try to keep the proof a secret, just because it feels like it would be devastating to a lot of people.
Robin Hanson: I don't think that's what he's getting at. Yes, surface similarities are correlated with structural similarities, or mathematical similarities (I know of a guy who found a couple big research papers towards his astrophysics PhD via a colleage's analogy between gravitational and electromagnetic waves), but they show up so often under other circumstances that it is meet to be suspicious of them. The outside view works really well for Christmas shopping, essay writing, program development, and the like because it is obvious that the structural similarities are present.
Joseph Knecht: When you say that you haven't seen evidence that puts "soul" on shaky grounds, [...]
Sorry, poor wording - please substitute "but nor have I seen evidence against 'choice' of the kind which puts 'soul' on shaky grounds." I am familiar with many of the neurological arguments against souls - I mentioned the concept because I am not familiar with any comparable evidence regarding choice. (Yes, I have heard of the experiments which show nervous impulses towards an action prior to the time when the actor thought they decided. That's interesting, but it's no Phineas Gage.)
Joseph Knecht: It is a clash of intuitions, then? I freely admit that I have seen no such account either, but nor have I seen the kind of evidence which puts "soul" on shaky grounds. And "fire" is comparably ancient to "soul", and still exists.
In fact, "fire" even suggests an intermediate position between yours and that which you reject: chemically, oxidation reactions like that of fire show up all over the place, and show that the boundary between "fire" and "not fire" is far from distinct. Would it be surprising were it found that the boundary between the prototypical human choices Eliezer names and your not-choices is blurry in a similar fashion?
Kip Werking, I can see where you're coming from, but "free will" isn't just some attempt to escape fatalism. Look at Eliezer's post: something we recognize as "free will" appears whenever we undergo introspection, for example. Or look at legal cases: acts are prosecuted entirely differently if they are not done of one's "free will", contracts are annulled if the signatories did not sign of their own "free will". We praise good deeds and deplore evil deeds that are done of one's own "free will". Annihilation of free will requires rebuilding all of these again from their very foundations - why do so, then, when one may be confident that a reasonable reading of the term exists?
Brandon Reinhart: Jack Thompson. (Fortunately, he's been disbarred, now, so maybe that particular vein of stupidity is getting tapped out.)