Compartmentalization does make it sound that you forget that eating meat is unethical when it's decision time.
Now, clearly animal suffering is something I don't really care about. But that doesn't mean I have any argument or foundation for believing that it is legitimately unimportant.
Do you need an argument for believing it's legitimately unimportant? Why not just say that it's an arbitrary taste? The same goes for altruism - other people may benefit more from your money than you do, but, you don't care nearly as much about them as you care about yourself. Utilitarianism says that's wrong, but why should you think that utilitarianism is correct?
As for diets, when someone develops habits that maintain a diet, it's because they believe that diet to be correct.
You are right that tastes are a deciding factor, but you're taking it too far. According to you it impossible to act unethically, and/or your personal ethics must be consistently determined by your actions. I can essentially behave entirely arbitrarily and to you I will be obeying my own true code of ethics.
A big part of what this site addresses is how humans are inconsistent, irrational, and self-deceiving and short-termist. Can we at least agree that there are moments when people take actions that are more inconsistent, irrational and self-deceiving and ...
Neal Stephenson's The Diamond Age takes place several decades in the future and this conversation is looking back on the present day:
I'm not sure if I agree with this characterization of the current political climate; in any case, that's not the point I'm interested in. I'm also not interested in moral relativism.
But the passage does point out a flaw which I recognize in myself: a preference for consistency over actually doing the right thing. I place a lot of stock--as I think many here do--on self-consistency. After all, clearly any moral code which is inconsistent is wrong. But dismissing a moral code for inconsistency or a person for hypocrisy is lazy. Morality is hard. It's easy to get a warm glow from the nice self-consistency of your own principles and mistake this for actually being right.
Placing too much emphasis on consistency led me to at least one embarrassing failure. I decided that no one who ate meat could be taken seriously when discussing animal rights: killing animals because they taste good seems completely inconsistent with placing any value on their lives. Furthermore, I myself ignored the whole concept of animal rights because I eat meat, so that it would be inconsistent for me to assign animals any rights. Consistency between my moral principles and my actions--not being a hypocrite--was more important to me than actually figuring out what the correct moral principles were.
To generalize: holding high moral ideals is going to produce cognitive dissonance when you are not able to live up to those ideals. It is always tempting--for me at least--to resolve this dissonance by backing down from those high ideals. An alternative we might try is to be more comfortable with hypocrisy.
Related: Self-deception: Hypocrisy or Akrasia?