I didn't really know how to explain. Why do people always talk about Boltzmann brains? Their probability is always so low that they are almost never the answer to any problem, no matter how abstract. Even if it could be argued that Boltzmann brains would outnumber regular brains, all the observations you could make to support that would presumably be coming from within your Boltzmann consciousness.