After all, the assumption there is that Boltzmann's model goes wrong in predicting that we are Boltzmann brains. We know that we're not, so this is a bad prediction. This piece of reasoning seems to be something you deny we can actually do.

That's because it is. We DON'T know that we're not Boltzman brains. There would be no possible way for us to tell.