If you know the following fact: "The other guy will cooperate iff I cooperate", even if you know nothing about the nature of the cause of the correlation, that's still a good enough reason to cooperate.
You ask yourself "If I defect, what will the outcome be? If I cooperate, what will the outcome be?" Taking into account the correlation, one then determines which they prefer. And there you go.
For example, imagine that, say, two AIs that were created with the same underlying archetecture (though possibly with different preferences) meet up. They also know the fact of their similarity. Then they may reason something like "hrmm... The same underlying algorithms running in me are running in my opponent. So presumably they are reasoning the exact same way as I am even at this moment. So whichever way I happen to decide, cooperate or defect, they'll probably decide the same way. So the only reasonably possible outcomes would seem to be 'both of us cooperate' or 'both of us defect', therefore I choose the former, since it has a better outcome for me. Therefore I cooperate."
In other words, what I chose is also lawful. That is, physics underlies my brain. My decision is not just a thing that causes future things, but a thing that was caused by past things. If I know that the same past things influenced my opponent's decision in the same way, then I may be able to infer "whatever sort of reasoning I'm doing, they're also doing, so..."
Or did I completely fail to understand your objection?
Related to: Practical Rationality Questionnaire
Here among this community of prior-using, Aumann-believing rationalists, it is a bit strange that we don't have any good measure of what the community thinks about certain things.
I no longer place much credence in raw majoritarianism: the majority is too uneducated, too susceptible to the Dark Arts, and too vulnerable to cognitive biases. If I had to choose the people whose mean opinion I trusted most, it would be - all of you.
So, at the risk of people getting surveyed-out, I'd like to run a survey on the stuff Anna Salamon didn't. Part on demographics, part on opinions, and part on the interactions between the two.
I've already put up an incomplete rough draft of the survey I'd like to use, but I'll post it here again. Remember, this is an incomplete rough draft survey. DO NOT FILL IT OUT YET. YOUR SURVEY WILL NOT BE COUNTED.
Incomplete rough draft of survey
Right now what I want from people is more interesting questions that you want asked. Any question that you want to know the Less Wrong consensus on. Please post each question as a separate comment, and upvote any question that you're also interested in. I'll include as many of the top-scoring questions as I think people can be bothered to answer.
No need to include questions already on the survey, although if you really hate them you can suggest their un-inclusion or re-phrasing.
Also important: how concerned are you about privacy? I was thinking about releasing the raw data later in case other people wanted to perform their own analyses, but it might be possible to identify specific people if you knew enough about them. Are there any people who would be comfortable giving such data if only one person were to see the data, but uncomfortable with it if the data were publically accessible?