A few things.

• Your inference only follows if you insert a "and only if" into the sentence you quote. Just like "if you do work for me then you're entitled to compensation" doesn't imply that you aren't also entitled to compensation under other circumstances, "if how I raise my kids affects you then you're entitled to a say in how I do it" doesn't imply that you aren't also entitled to a say under other circumstances as well. Your willingness to jump so quickly to an unjustified inference suggests to me that you're projecting a context onto my post that I didn't put there. That sort of thing can cause a lot of misunderstandings; I encourage you to slow down a little accordingly.

• I have lots of problems with you subjecting your kids to horrible agonizing torture.

• Many of those problems are emotional and visceral. I don't think you have any obligation to take my emotional reactions to your child-raising practices into account. (Well, except in the very tenuous sense that those reactions do incur some very marginal costs on my part, but in practice that's lost in the noise.)

• If I disregard my emotional problems, and I disregard all cases where there is an effect on me, and I ask what's left over, I conclude that the value of the world (using my valuation, 'cuz who else's would I use?) with more tortured people in it is lower than with fewer. There is something to be said here about how that gives me a basis for action, but that's rather beside my original point.

• It also suggests that I have a basis for action to prevent you from torturing yourself when your doing so doesn't affect me at all, which is relevant to (and runs counter to) my original point. That's basically why I say I'm not comfortable with my original point. But I'm also not comfortable with saying you're entitled to forcibly keep me alive just because you think I'm better off alive than dead (or entitled to kill me if you think I'm better off dead than alive), so I don't think that situation is particularly simple or easy to defend.

affects

The trick is that the principle is sound, but those implications don't follow, because if I mind if someone does something, it thereby affects me.

One might ask: if I don't know something, how can it affect me, for "we care only about our own states of mind"?

As it turns out, I care about the world directly, and that's the meaning of "affect" here - affecting my utility, not affecting my perception of my utility.