I accept your working definitions for "internal evidence" and "external evidence."
When we are attempting to describe an individual's epistemic status -- the amount of reliable information they possess about the world -- then it seems that external evidence is the relevant variety of evidence to consider
I want to be a little careful about the words "epistemic status" and "reliable information," because a lot of confusion can be introduced through the use of terms that abstract.
I remember reading once that courtship behavior in robins is triggered by the visual stimulus of a patch of red taller than it is wide. I have no idea if this is actually true, but suppose it is. The idea was that the ancestral robin environment didn't contain other stimuli like that other than female robins in estrus, so it was a reliable piece of evidence to use at the time. Now, of course, there are lots of visual stimuli in that category, so you get robins initiating courtship displays at red socks on clotheslines and at Coke cans.
So, OK. Given that, and using your terms, and assuming it makes any sense to describe what a robin does here as updating on evidence at all, then a vertical red swatch is always internal evidence of a fertile female, and it was external evidence a million years ago (when it "genuinely" counterfactually depended on the presence of such a female) but it is not now. If we put some robins in an environment from which we eliminate all other red things, it would be external evidence again. (Yes?)
If what I am interested in is whether a given robin is correct about whether it's in the presence of a fertile female, external evidence is the relevant variety of information to consider.
If what I am interested in is what conclusions the robin will actually reach about whether it's in the presence of a fertile female, internal evidence is the relevant variety of information to consider.
If that is consistent with your claim about the robin's epistemic status and about the amount of reliable information the robin possesses about the world, then great, I'm with you so far. (If not, this is perhaps a good place to back up and see where we diverged.)
if two observers differ substantially in the external evidence available to them, it seems justifiable to place them in separate reference classes for certain anthropic explanations.
Sure, when available external evidence is particularly relevant to those anthropic explanations.
If the superintelligence were engaging in anthropic reasoning, should it put itself in the same reference class as the mental patients in all cases?
So A and B both believe they're superintelligences. As it happens, A is in fact a SI, and B is in fact a mental patient. And the question is, should A consider itself in the same reference class as B. Yes?
...I think the answer is fairly obviously no, and this is because of the vast difference in the epistemic situations of the superintelligence and the mental patients,
Absolutely agreed. I don't endorse any decision theory that results in A concluding that it's more likely to be a mental patient than a SI in a typical situation like this, and this is precisely because of the nature of the information available to A in such a situation.
If we think identical (or similar) internal evidence requires that they be in the same reference class, then I think the answer may be yes.
Wait, what?
Why in the world would A and B have similar internal evidence?
I mean, in any normal environment, if A is a superintelligence and B is a mental patient, I would expect A to have loads of information on the basis of which it is procedurally rational for A to conclude that A is in a different reference class than B. Which is internal evidence, on your account. No?
But, OK. If I assume that A and B do have similar internal evidence... huh. Well, that implicitly assumes that A is in a pathologically twisted epistemic environment. I have trouble imagining such an environment, but the world is more complex than I can imagine. So, OK, sure, I can assume such an environment, in a suitably hand-waving sort of way.
And sure, I agree with you: in such an environment, A should consider itself in the same reference class as B. A is mistaken, of course, which is no surprise given that it's in such an epistemically tainted environment.
Now, I suppose one might say something like "Sure, A is justified in doing so, but A should not do so, because A should not believe falsehoods." Which would reveal a disconnect relating to the word "should," in addition to everything else. (When I say that A should believe falsehoods in this situation, I mean I endorse the decision procedure that leads to doing so, not that I endorse the result.)
But we at least ought to agree, given your word usage, that it is procedurally rational for A to conclude that it's in the same reference class as B in such a tainted environment, even though that isn't true. Yes?
Summary: There are claims that Boltzmann brains pose a significant problem for contemporary cosmology. But this problem relies on assuming that Boltzmann brains would be part of the appropriate reference class for anthropic reasoning. Is there a good reason to accept this assumption?
Nick Bostrom's Self Sampling Assumption (SSA) says that when accounting for indexical information, one should reason as if one were a random sample from the set of all observer's in one's reference class. As an example of the scientific usefulness of anthropic reasoning, Bostrom shows how the SSA rules out a particular cosmological model suggested by Boltzmann. Boltzmann was trying to construct a model that is symmetric under time reversal, but still accounts for the pervasive temporal asymmetry we observe. The idea is that the universe is eternal and, at most times and places, at thermodynamic equilibrium. Occasionally, there will be chance fluctuations away from equilibrium, creating pockets of low entropy. Life can only develop in these low entropy pockets, so it is no surprise that we find ourselves in such a region, even though it is atypical.
The objection to this model is that smaller fluctuations from equilibrium will be more common. In particular, fluctuations that produce disembodied brains floating in a high entropy soup with the exact brain state I am in right now (called Boltzmann brains) would be vastly more common than fluctuations that actually produce me and the world around me. If we reason according to SSA, the Boltzmann model predicts I am one of those brains and all my experiences are spurious. Conditionalizing on the model, the probability that my experiences are not spurious is minute. But my experiences are in fact not spurious (or at least, I must operate under the assumption that they are not if I am to meaningfully engage in scientific inquiry). So the Boltzmann model is heavily disconfirmed. [EDIT: As AlexSchell points out, this is not actually Bostrom's argument. The argument has been made by others. Here, for example.]
Now, no one (not even Boltzmann) actually believed the Boltzmann model, so this might seem like an unproblematic result. Unfortunately, it turns out that our current best cosmological models also predict a preponderance of Boltzmann brains. They predict that the universe is evolving towards an eternally expanding cold de Sitter phase. Once the universe is in this phase, thermal fluctuations of quantum fields will lead to an infinity of Boltzmann brains. So if the argument against the original Boltzmann model is correct, these cosmological models should also be rejected. Some people have drawn this conclusion. For instance, Don Page considers the anthropic argument strong evidence against the claim that the universe will last forever. This seems like the SSA's version of Bostrom's Presumptuous Philosopher objection to the Self Indication Assumption, except here we have a presumptuous physicist. If your intuitions in the Presumptuous Philosopher case lead you to reject SIA, then perhaps the right move in this case is to reject SSA.
But maybe SSA can be salvaged. The rule specifies that one need only consider observers in one's reference class. If Boltzmann brains can be legitimately excluded from the reference class, then the SSA does not threaten cosmology. But Bostrom claims that the reference class must at least contain all observers whose phenomenal state is subjectively indistinguishable from mine. If that's the case, then all Boltzmann brains in brain states sufficiently similar to mine such that there is no phenomenal distinction must be in my reference class, and there's going to be a lot of them.
Why accept this subjective indistinguishability criterion though? I think the intuition behind it is that if two observers are subjectively indistinguishable (it feels the same to be either one), then they are evidentially indistinguishable, i.e. the evidence available to them is the same. If A and B are in the exact same brain state, then, according to this claim, A has no evidence that she is in fact A and not B. And in this case, it is illegitimate for her to exclude B from her anthropic reference class. For all she knows, she might be B!
But the move from subjective indistinguishability to evidential indistinguishability seems to ignore an important point: meanings ain't just in the head. Even if two brains are in the exact same physical state, the contents of their representational states (beliefs, for example) can differ. The contents of these states depend not just on the brain state but also on the brain's environment and causal history. For instance, I have beliefs about Barack Obama. A spontaneously congealed Boltzmann brain in an identical brain state could not have those beliefs. There is no appropriate causal connection between Obama and that brain, so how could its beliefs be about him? And if we have different beliefs, then I can know things the brain doesn't know. Which means I can have evidence the brain doesn't have. Subjective indistinguishability does not entail evidential indistinguishability.
So at least this argument for including all subjectively indistinguishable observers in one's reference class fails. Is there another good reason for this constraint I haven't considered?
Update: There seems to be a common misconception arising in the comments, so I thought I'd address it up here. A number of commenters are equating the Boltzmann brain problem with radical skepticism. The claim is that the problem shows that we can't really know we are not Boltzmann brains. Now this might be a problem some people are interested in. It is not one that I am interested in, nor is it the problem that exercises cosmologists. The Boltzmann brain hypothesis is not just a physically plausible variant of the Matrix hypothesis.
The purported problem for cosmology is that certain cosmological models, in conjunction with the SSA, predict that I am a Boltzmann brain. This is not a problem because it shows that I am in fact a Boltzmann brain. It is a problem because it is an apparent disconfirmation of the cosmological model. I am not actually a Boltzmann brain, I assure you. So if a model says that it is highly probable I am one, then the observation that I am not stands as strong evidence against the model. This argument explicitly relies on the rejection of radical skepticism.
Are we justified in rejecting radical skepticism? I think the answer is obviously yes, but if you are in fact a skeptic then I guess this won't sway you. Still, if you are a skeptic, your response to the Boltzmann brain problem shouldn't be, "Aha, here's support for my skepticism!" It should be "Well, all of the physics on which this problem is based comes from experimental evidence that doesn't actually exist! So I have no reason to take the problem seriously. Let me move on to another imaginary post."