This seems to be really hitting on an issue that is only marginally related to Boltzmann brains and is made more confusing by the really counterintuitive stuff about Boltzmann brains.

Whenever one is trying to make any anthropic argument one has to ask what is an observer? If one believed in ontologically irreducible observers (something close to the classical notion of a soul in many cultures) this wouldn't be a problem. The problem here arises primarily from the difficulty in trying to understand what it means for something to be an observer in a universe where no observer seems to be irreducible.

Incidentally, I sometimes think that the Boltzmann brain argument is an argument that something is very wrong with our understanding of the eventual fate of the universe. The essential problem is that the idea doesn't add up to normality. I don't know how much this should impact my estimates at all (such as whether it should make me slightly doubt current estimates that say we won't have a Big Crunch). Anthropics can be really confusing.