Yes, though depending on your (definitive or provisional) conclusions about how much sapience matters, there may be an inflection point.

At the bottom of that scale, I wouldn't worry about eating very small animals because very small brains seem to make for negligible amounts of moral concern. At the higher end, and as this link from elsewhere in this thread suggests, larger animals are more "suffering efficient" to coin a phrase both horrible and awkard, but also suggestive.

I don't think an oyster suffers in any meaningful sense, and I don't worry a whole lot about fish. I worry more about chickens and hogs than about cows because it takes a larger number of them to yield an equivalent mass of meat.