Yes, though depending on your (definitive or provisional) conclusions about how much sapience matters, there may be an inflection point.
At the bottom of that scale, I wouldn't worry about eating very small animals because very small brains seem to make for negligible amounts of moral concern. At the higher end, and as this link from elsewhere in this thread suggests, larger animals are more "suffering efficient" to coin a phrase both horrible and awkard, but also suggestive.
I don't think an oyster suffers in any meaningful sense, and I don't worry a whole lot about fish. I worry more about chickens and hogs than about cows because it takes a larger number of them to yield an equivalent mass of meat.
I have noticed that among philosophers, vegetarianism of one form or another is quite common. In fact, I became a vegetarian (technically a pescetarian) myself partly out of respect for an undergraduate philosophy professor. I am interested in finding out if there is a similar disproportion in the Less Wrong community.
I didn't request that this go into Yvain's survey because I want more information than just what animal products you do or don't eat; I'd also like to see nuances of the reasons behind your diet. There are a lot more shades than carnivore/vegetarian/vegan - if you want to be a vegetarian but are allergic to soy and gluten, that's a compelling reason to diversify protein sources, for instance. I'd also like to hear about if you avoid any plant foods (if you think they're farmed in a way that's environmentally destructive or that hurts people or if you have warm fuzzy feelings for plants, maybe). Here are some questions that come to mind: