It doesn't seem at all clear that if (say) you are upset by the look of my shirt (caused by light from my shirt hitting your retina) that I have presumptively wronged you.

That would be a case of indirect causation of suffering, as I discussed above with the example of atheistic speech upsetting someone. I'm not sure exactly what the direct/indirect distinction amounts to, but, in practice, I don't think people usually have a very hard time with the distinction. Some features that might be relevant: the upset in the ugly shirt case and the atheistic speech case are both cognitively mediated, and neither of them is biologically hardwired.

The direction of matter/energy transfer is relevant because it distinguishes something I do to you, from something that just happens. If a hot sunny day overheats you, that just happens. If I burn you with a laser, that is something I did to you. Similar points apply to stabbings, shootings, poisonings, but with matter instead of energy becoming the weapon. In most people's moral views, the action / mere-happening distinction is important.