causation might be in the map rather than the territory
Of course it is. There is no atom of causation anywhere. It's a tool for embedded agents to construct useful models in an internally partially predictable universe.
"Backward causation" may or may not be a useful model at times, but it is certainly nothing but a model.
As a trained (though not practicing) physicist, I can see that you are making a large category error here. Relativity neither adds to not subtracts from the causation models. In a deterministic Newtonian universe you can imagine backward causation as a useful tool. Sadly, its usefulness it rather limited. For example, the diffusion/heat equation is not well posed when run backwards, it blows up after a finite integration time. An intuitive way to see that is that you cannot reconstruct the shape of a glass of water from the puddle you see on the ground some time after it was spilled. But in cases where the relevant PDEs are well posed in both time directions, backward causality is equivalent to forward causality, if not computationally, then at least in principle.
All that special relativity gives you is that the absolute temporal order of events is only defined when they are within a lightcone, not outside of it. General relativity gives you both less and more. On the one hand, the Hilbert action is formulated without referring to time evolution at all and poses no restriction on the type of matter sources, be they positive or negative density, subluminal or superluminal, finite or singlular. On the other hand, to calculate most interesting things, one needs to solve the initial value problem, and that one poses various restrictions on what topologies and matter sources one can start with. On the third hand, there is a lot of freedom to define what constitutes "now", as many different spacetime foliations are on equal footing.
If you add quantum mechanics to the mix, the Born rule, needed to calculate anything useful regardless of one's favorite interpretation, breaks linearity and unitarity at the moment of interaction (loosely speaking) and is not time-reversal invariant.
The entropic argument is also without merit: there is no reason to believe that entropy would decrease in a "high-entropy world", whatever that might mean. We do not even know how observer-independent entropy is (Jaynes argued that apparent entropy depends on the observer's knowledge of the world).
Basically, you are confusing map and territory. If backward causality helps you make more accurate maps, go wild, just don't claim that you are doing anything other than constructing models.
I think we can go a bit farther in predicting that backwards causation will be a useful concept in some very specific cases, which will break down far above the scale of the normal second law.
We "see" backwards causation when we know the outcome but not how the system will get there. What does this behavior sound like a hallmark of? Optimization processes! We can predict in advance that backwards causation will be a useful idea to talk about the behavior of some optimization processes, but that it will stop contributing useful information when we want to zoom in past the "intentional stance" level of description.
I feel that questions like this have a hard time escaping confusion because the notion of linear time is so deeply associated with causality already.
Could you point me to the arguments about a high-entropy universe being expected to decrease in entropy?
I don't think backwards causation is absurd, more or less for the reasons you sketch. Another minor reason: Some philosophers like "effective strategy" accounts of causation, according to which we define causation via its usefulness for agents trying to achieve goals. On these accounts, backwards causation is pretty trivial--just suppose you live in a deterministic universe and your goal is to "make the state of the universe at the Big Bang such that I eat breakfast tomorrow." The philosopher Gary Drescher argues something similar in Good and Real if I recall correctly.
That said, I don't think we are really explaining or de-confusing anything if we appeal to backwards causation to understand Newcomb's Problem or argue for a particular solution to it.
"That said, I don't think we are really explaining or de-confusing anything if we appeal to backwards causation to understand Newcomb's Problem or argue for a particular solution to it." - How come?
Perhaps I was too hasty. What I had in mind was the effective strategy strategy--if you define causation by reference to what's an effective strategy for achieving what, then that means you are assuming a certain decision theory in order to define causation. And so e.g. one-boxing will cause you to get a million if EDT is true, but not if CDT is true.
If instead you have another way to define causation, then I don't know. But for some ways, you are just fighting the hypothetical--OK, so maybe in the original Newcomb's Problem as stated, backwards causation saves the day and makes CDT and EDT agree on what to do. But then what about a modified version where the backwards causation is not present?
Another relativistic argument against time flowing is that simultaneity is only defined relative to a reference frame. Therefore, there is no unified present which is supposed to be what is flowing.
Relativity does not make the arrow of time relative to observer. Events in one's future light cone remain in their future light cone also from a perspective of someone else.
In Newcomb's problem an agent picks either one-box or two-box and finds that no matter which option they picked, a predictor predicted them in advance. I've gone to a lot of effort to explain how this can be without requiring backwards causation (The Prediction Problem, Deconfusing Logical Counterfactuals), yet now I find myself wondering if backwards causation is such a bad explanation after all.
Unfortunately I'm not a physicist, so take what I say with a grain of salt, but I'll sketch out some reasons why backwards causation might not be as ridiculous as it first seems and hopefully someone else develops this in more detail.
One prominent theory of time is Eternalism in which there is no objective flow of time and terms like "past", "present" and "future" can only be used in a relative sense. An argument in favour of this is that it is often very convenient in physics to model space-time as a 4-dimensional space. If time is just another dimension, why should the future be treated differently than the past? Nothing in this model differentiates the two. If we have two blocks X and Y next to each other, we can view either X as the left one or Y as the left one depending on the direction we look at it from. Similarly, if A causes B in the traditional forwards sense, why can't we symmetrically view B as backwards causing A, where again if we viewed it another way A to B would be backwards causation and B to A would be forwards causation.
Another relativistic argument against time flowing is that simultaneity is only defined relative to a reference frame. Therefore, there is no unified present which is supposed to be what is flowing. This doesn't mean that the universe couldn't be described by a unidirectional graph. However, it does greatly undermine any trust in our naive intutions related to time.
Thirdly, entropy has often been the arrow of time with other physical laws claimed to be reversible. We are in a low-entropy world so entropy increases. However, if we were in a high-entropy world, it would decrease, so time and causation would seem to be going backwards (from our perspective). This would seem to suggest that backwards causation is just as valid a phenomenon as backward causation.
I want finish by reminding readers again that I am not a physicist. This post is more intended to spark discussion that anything else.
(Another possibility I haven't discussed is that causation might be in the map rather than the territory)