One other quick remark.

We have talked about it quite a bit, and I don't believe that we're a cult. However, in every conversation I play out in my head where I try to talk about what we're doing here with someone who's not part of it, they start to think we're a cult within about thirty seconds. I'm almost thinking of using "I've joined a cult" as my opening line, to get it out of the way.

I think you may be expecting too much from us. We're not mostly trained philosophers or psychologists or neurobiologists. We're not even self-trained supermen who can break holes in brick walls with sheer brainpower. We're mostly computer programmers who are looking for something else to read when we should be working.

Experiments can always be framed as a waste of resources.

There is always something you're using up that you could put to direct productive use, even if it's just your time.

Did you write a cost function down for the various debate outcomes? The skew will inform whether overconfidence or underconfidence should be weighted differently.

Or is your point that once a 'good' label is assigned we just increment its goodness index and forget the detailed reasoning that led us to it?

Exactly that. We may be able to recall our reasoning if we try to, but we're likely to throw in a few extra false justifications on top, and to forget about the other side.

pizza is good, seafood is bad

When I say something is good or bad ("yay doggies!") it's usually a kind of shorthand:

pizza is good == pizza tastes good and is fun to make and share

seafood is bad == most cheap seafood is reprocessed offcuts and gave me food poisoning once

yay doggies == I find canine companions to be beneficial for my exercise routine, useful for home security and fun to play with.

I suspect when most people use the words 'good' and 'bad' they are using just this kind of linguistic compression. Or is your point that once a 'good' label is assigned we just increment its goodness index and forget the detailed reasoning that led us to it? Sorry, the post was an interesting read but I'm not sure what you want me to conclude.

At the Go club, some-one asked about using red, green, and blue stones instead of using black and white. The chap who is doing a PhD in game theory said: the two weakest players will gang up on the strongest player, *just like any truel".

I was surprised by the way he spoke immediately without being distracted from his own game. Study long enough and hard enough and it becomes automatic: gang up on the stronger.

Now humans have an intuitive grasp of social games, which raises the question: what would that algorithm feel like from the inside? Perhaps it gets expressed as sympathy for the underdog?

It might be possible to test this hypothesis. A truel is a three player game that turns into a duel after one player has been eliminated. That is why you side with the weaker of your two opponents. The experimental psychologist setting up his experiment can manipulate the framing. It the game theory idea is correct, sympathy for the underdog should be stronger when the framing primes the idea of a follow on duel.

For example if you frame America versus bin Laden as the battle of two totalising ideologies, will the world be dog-eat-dog Capitalist or beard-and-burka Islamic, that should boost underdog-sympathy. If you frame America versus bin Laden as pure tragedy, "Americans just want to stay at home eating, bin Laden really wanted to stay in Mecca praying, how came they ended up fighting?", that should weaken underdog sympathy.

I'm not sure how to set up such an experiment. May it could be presented as research into writing dialogue for the theatre. The experimental subject is presented with scripts for plays. In one play the quarreling characters see rivals they are discussing (e.g. Israel v Palestine, etc) as expansionist, in another play the quarreling characters see the rivals they are discussing as fated to fight then go home. The experimental subject is probed with various lines of dialogue for the characters, which either sympathise with the underdog or the overdog, and asked to judge which seems natural.

The hypothesis is that it is the characters that anticipate a follow on duel whose dialogue feels natural with sympathy for the underdog.

In the old days science was done by independently wealthy people, and how successful it was. Can today's scientists create profitable enterprises to fund their work? FHI is supposedly full of good programmers. Say, everyone works on a joint project half a week and does science the other half. Sounds like a real world application for the community's rationality and cooperation abilities, what do you say Eliezer?

By definition someone for whom religion or spirituality is intensely personal is going to avoid talking to you about it. The fact that that in all the conversations about religion you have ever had, no-one has declined to participate on those grounds is hardly evidence that these people don't exist.

Hmmm, methinks you are moderately wrong about religious organisations being on the wrong side of 'every' moral issue in American history. You've heard of the Quakers - funny hats, oatmeal, social justice and all that.

I just don't see modern secular churches (ok so maybe you don't have those in the USA yet...) like the Anglicans as a major force for irrationality. When they bump up against science there are a few protests and then they cede ground, and explain any contradictions between scripture and reality by admitting scripture is mostly just stories.

Get them reading. Babies love being read to. Introduce them to the beauty of books, and the wonders of the public library system. Then, when they have the tools to navigate the repository of written knowledge, set them loose. Steer a little, but don't interfere.