If they're a truly isolated hermit, that distinction would presumably no longer apply, but the world is pretty short on truly isolated hermits.

I think you probably could kill and replace an isolated hermit in a QALY-neutral way (you'd probably need a fairly unhappy person to keep it QALY neutral even,) whereas with social connections in the equation, if you were trying to kill and replace non-hermits in a QALY neutral way, you'd ultimately end up having to do it to everyone.