Current physics models get around that question anyways. The way our brains work, there is more entropy after a memory is burned than before. Thus, time seems to flow from low to high entropy to us. If entropy was flowing the another direction, than our brains would think of another direction as past. The laws of thermodynamics are a side effect of how our brains process time.

Thus we can have low entropy -> high entropy without a shit ton of Boltzmann Brains.