Your claim that harms adversely impact reproduction is controversial because of the obvious counter examples. X has already lost their ability to reproduce. X may not even care about reproducing. It's still harmful to X to strap him down and torture him. Therefore X can be harmed without adversely affecting his ability to reproduce. This is not to mention the imaginable beings who can undergo suffering (and this is not a claim that harm reduces to suffering alone, but that suffering is a kind of harm), but which were not built by natural selection, and which don't have biology even remotely related to reproducing. There are also counter examples going the other way, equally as obvious. Y doesn't want to have children. She uses a contraception. She avoids pregnancy. Her ability to reproduce has been impeded, but she quite clearly has not been harmed (indeed, she has been liberated from the shackles of biology by glorious technology).

I don't think anyone would claim that adverse affects to reproductive ability are completely orthogonal with harm - perhaps a decrease in your ability to reproduce would be more likely to be harmful than not - but to be honest, it doesn't even look to me like the correlation's all that strong. What seems downright obvious, though, is that one does not reduce to the other.

Also, natural selection doesn't care about whether I can reproduce. It's not a caring-type thing. It's an optimisation process which doesn't make use of caring at any point during the process, and I would only care about what natural selection selects if it were important to me to be naturally selected. And why would it be? Producing APMason-like forms is not even close to being my biggest concern.