The capacity to abide by morality carves out the right cluster in thingspace for me, though I'd hesitate to call it the determining factor. If a thing has this capacity, we care about its preferences proportionately.
People, save probably infants, are fully capable, in theory, of understanding and abiding by morality. Most animals are not. Those more capable of doing so, domestic pets, beasts of burden for example, receive some protection. Those who do not have this capacity are generally less protected and that which is done to them less morally relevant.
I don't fully endorse this view, but it feels like it explians a lot.
This is a logical vicious circle. Morality itself is the handmaiden of humans (and similar creatures in fantasy and SF). Morality has value only insofar as we find it important to care about human and quasi-human interests. This does not answer the question "Why do we care about human and quasi-human interests?"
One could try to find an answer in the prisoner's dilemma. In the logic of Kant's categorical imperative. Cooperation of rational agents and the like. Then I should sympathize with any system that cares about my interests, even if that system is otherwise like the Paperclipmaker and completely devoid of "unproductive" self-reflection. Great. There is some cynical common sense in this, but I feel a little disappointed.
For a long time, I wanted to ask something. I was just thinking about it again when I saw that Alicorn has a post on a similar topic. So I decided to go ahead.
The question is: what is the difference between morally neutral stimulus responces and agony? What features must an animal, machine, program, alien, human fetus, molecule, or anime character have before you will say that if their utility meter is low, it needs to be raised. For example, if you wanted to know if lobsters suffer when they're cooked alive, what exactly are you asking?
On reflection, I'm actually asking two questions: what is a morally significant agent (MSA; is there an established term for this?) whose goals you would want to further; and having determined that, under what conditions would you consider it to be suffering, so that you would?
I think that an MSA would not be defined by one feature. So try to list several features, possibly assigning relative weights to each.
IIRC, I read a study that tried to determine if fish suffer by injecting them with toxins and observing whether their reactions are planned or entirely instinctive. (They found that there's a bit of planning among bony fish, but none among the cartilaginous.) I don't know why they had to actually hurt the fish, especially in a way that didn't leave much room for planning, if all they wanted to know was if the fish can plan. But that was their definition. You might also name introspection, remembering the pain after it's over...
This is the ultimate subjective question, so the only wrong answer is one that is never given. Speak, or be wrong. I will downvote any post you don't make.
BTW, I think the most important defining feature of an MSA is ability to kick people's asses. Very humanizing.