People nearer the front think that they have the moral right to get off earlier than people behind them, regardless of whether they got their seat through choice or chance. People also like to get off with the other members of their party.

So people nearer the front will defect from this solution even though all but the first half dozen rows would probably be better off cooperating. Once all the people in front of passenger X have gotten off, passenger X will defect as well.

I'm seldom in a hurry to get off the plane (I know there's just more waiting once you're off) so I wait till there are gaps in traffic to get out of my seat and retrieve my luggage. Of course I can only get away with this if I have my preferred window seat. Otherwise, in deference to the greedy (but conventional) expectations of the people I'm trapping next to me, I have to get off as quickly as I'm able.

Contrary to your usual practice of including voluminous relevant links, you didn't point to anything specific for Judea Pearl. Let's give this link for his book Causality, which is where people will find the graphical calculus you rely on.

You've mentioned Pearl before, but haven't blogged the details. Do you expect to digest Pearl's graphical approach into something OB-readers will be able to understand in one sitting at some point? That would be a real service, imho.

I've traveled in Europe, and seen remnants of the roman roads, walls and viaducts. One of the .sigs I use most often is this:

C. J. Cherryh, "Invader", on why we visit very old buildings: "A sense of age, of profound truths. Respect for something hands made, that's stood through storms and wars and time. It persuades us that things we do may last and matter."

Thinking about your declaration "If you run around inspecting your foundations, I expect you to actually improve them", I now see that I've been using "PCR" to refer to the reasoning trick that Bartley introduced (use all the tools at your disposal to evaluate your foundational approaches) to make Pan-Critical Rationalism an improvement over Popper's Critical Rationalism. But, for Bartley, PCR was just a better foundation for the rest of Popper's epistemology, and you would replace that epistemology with something more sophisticated. For me, the point of emphasizing PCR is that you should want Bartley's trick as the unchangable foundation below everything else.

If an AI is going to inspect its foundations occasionally, and expect to be able to improve on them, you'd better program it to use all the tools at its disposal to evaluate the results before making changes. This rule seems more fundamental than guidelines on when to apply Occam, induction, or Bayes rule.

If Bartley's trick is the starting point, I don't know whether it would be necessary or useful to make that part of the code immutable. In terms of software simplicity, not having a core that follows different principals would be an improvement. But if there's any chance that the AI could back itself into a corner that would lead it to conclude that there were a better rule to decide what tools to rely on, everything might be lost. Hard-coding Bartley's trick might provide the only platform to stand on that would give the AI a way to rebuild after a catastrophe.

I now understand the reluctance to call the result PCR: it's not the whole edifice that Bartley (& Popper) constructed, you only use the foundation Bartley invented.

Hurrah! Eliezer says that Bayesian reasoning bottoms out in Pan-Critical Rationalism.

re: "Why do you believe what you believe?"

I've always said that Epistemology isn't "the Science of Knowledge" as it's often called, instead it's the answer to the problem of "How do you decide what to believe?" I think the emphasis on process is more useful than your phrasing's focus on justification.

BTW, I don't disagree with your stress on Bayesian reasoning as the process for figuring out what's true in the world. But Bartley really did successfully provide the foundation for rational analysis. When you want to figure out how to think successfully, you should use all the tools at your disposal (pan-critical) because at that point, you shouldn't be taking anything for granted.

@Wes: "This doctrine still leaves me wondering why this meta-level hermeneutic of suspicion should be exempt from its own rule." It's not exempt. Read "The Retreat to Commitment" by W. W. Bartley III. There's a substantial section in which Bartley presents the best arguments he can find against Popper's Epistemology (and WWB's fix to it) and shows how the criticisms come up short. Considering your opponent's best arguments is an important part of the process.

@Peter Turney: I like your description of "incremental doubt" because it illustrates how Bartley was saying that none of your beliefs has to be foundational. You should examine each of them in turn, but you have to find a different place to stand for each of those investigations.

1. Fund the top half of The Copenhagen Consensus projects. 2. Longevity research: Give a billion to Aubrey de Grey. 3. Push the US government towards more support of liberty. Money on that scale could make a significant start to unwinding the welfare state. a. The Institute for Justice has a very good program making practical steps. They could productively spend at least 10 times their current budget. Think about whether their methods can be applied in other areas. b. Try to convince Marshall Fritz to return to the Advocates for Self Government. He pioneered a process of inventing tools to spread liberty, and then measure the results to decide how to spend more money. c. Start think tanks to flood the political market with arguments and (funded) proposals for moving toward liberty. The Cato Institute does a good job, but in this case, I'd expect to improve things more by providing them with competition than with funding. 4. Buy OLPCs for the kids in all the "bottom billion" countries.

Patrick, that was my interpretation. I had time to come up with one proposal. (I'm not able to commit full-time to being a student of bayescraft at this point.)

Z. M. Davis, thanks for the pointer.

There's a particular kind of groupthink peculiar to scholarly fields. In my review of "The Trouble with Physics", I pointed to two (other) specific examples of recent advances that were stymied for long periods of time by scholarly groupthink. There are many others.

But I think Eli has hit on another important mechanism. Few learners these days are expected to rediscover important concepts, so we get no training in this ability. I don't see how turning scientific knowledge into a body of secrets will address the problem, but it's a valuable insight. I'd offer solving puzzles and breaking codes as alternative training for finding the patterns that nature is hiding from us. More scientists should spend their time entering puzzle contests, hunting geocaches, and attacking cryptosystems.

And could someone provide an interpretation of the cast of characters here? I enjoyed the list that was presented for a previous article.

"... the overwhelming majority might as well belong to a religious cargo cult based on the notion that self-modifying AI will have magical powers."

"Maybe you can admire someone who directly thinks you're a crackpot, but I can't."

I have a high regard for most extropians (a subset of Transhumans, I think) I know well, but that doesn't make me believe that the Egan line is more than hyperbole at most. I don't take it as a slur against anyone whose name I know. I've certainly seen evidence that the majority wouldn't be able to distinguish the magical explanations that appear.

And the fact that Charles Stross thinks that discussing Extropianism is attractive to his market makes me think Egan has more truth on his side.

But I also want to mention Egan's "Diaspora". I bring it often as a great fictional depiction of an AI awakening. I know, I know. "Arguing from fictional evidence." But many people expect coming to awareness to be magic, and Egan shows how it could happen in a step-by-step manner.

Eliezer, that was just beautiful.

"Rest assured that you are not holding the mere appearance of a banana. There really is a banana there, not just a collection of atoms."