(for example, if we killed all the predators there would be lots more prey animals, but if they tend to have lives that are on average worse than not living at all then this would be a bad thing.)

Wouldn't that logic lead you to killing all predators or all prey depending on the answer to the question of whether the prey has lives not worth living? If "no", kill prey, if "yes", kill predators. In any case you're committed to a lot of killing.