It doesn't seem at all clear that if (say) you are upset by the look of my shirt (caused by light from my shirt hitting your retina) that I have presumptively wronged you.

That would be a case of indirect causation of suffering, as I discussed above with the example of atheistic speech upsetting someone. I'm not sure exactly what the direct/indirect distinction amounts to, but, in practice, I don't think people usually have a very hard time with the distinction. Some features that might be relevant: the upset in the ugly shirt case and the atheistic speech case are both cognitively mediated, and neither of them is biologically hardwired.

The direction of matter/energy transfer is relevant because it distinguishes something I do to you, from something that just happens. If a hot sunny day overheats you, that just happens. If I burn you with a laser, that is something I did to you. Similar points apply to stabbings, shootings, poisonings, but with matter instead of energy becoming the weapon. In most people's moral views, the action / mere-happening distinction is important.

There's a combination of

1. A defeasible presumption that if I project matter and/or energy into you, and that directly causes you to suffer, I'm in the moral wrong.
2. Ideas about what counts as projecting matter and energy that do not follow the literal definition of projecting matter and energy.

People don't think of seeing a shirt as the shirt doing something to their retina. They think of it as them reacting to a shirt and the shirt just sitting there. The fact that you see the shirt by reflected light is an irrelevant technical detail. If the shirt was illuminated by a lamp on your front lawn instead of by the sun, the technical details would change (since the light is coming from you) but the answer to who is wrong would not change.