"Causality is based on entropy increase, so it can only make sense to draw causal arrows “backwards in time,” in those rare situations where entropy is not increasing with time. [...] where physical systems are allowed to evolve reversibly, free from contact with their external environments." E.g. the normal causal arrows break down for, say, CMB photons. -- Not sure how Scott jumps from reversible evolution to backward causality.

It's a few paragraphs up, where he says:

Now, the creation of reliable memories and records is essentially always associated with an increase in entropy (some would argue by definition). And in order for us, as observers, to speak sensibly about “A causing B,” we must be able to create records of A happening and then B happening. But by the above, this will essentially never be possible unless A is closer in time than B to the Big Bang.

That is, we are only capable of remembering (by any means) things closer to the Big Bang, because memories require entropy increase; and furthermore, memories are necessary for drawing a causal arrow that orders past vs future. But if there is a system that stays isentropic, it needn't have such a ordering.

Note: this is actually very close to Drescher's resolution of Loschmidt's paradox ("why is physics time-symmetric but entropy isn't?") in Good and Real: since entropy determines what we (or any observers) regard as pastward, we will necessarily observe only those time histories of increasing entropy.