"But you'd have to give up the link between causality and what we call 'time'."

You'd just have to make it slightly weaker. Entropy will still by and large increase in the direction we call "forward in time". So long as entropy is increasing, causality works. I don't think the errors would be enough to notice in any feasible experiment.