"I would let the five die and not feel guilty about it, because I am not the cause of their deaths."
A more charitable way of phrasing the consequentialist PoV here is that we care more about stopping the deaths than avoiding feelings of guilt. Yes, it's true that on certain accounts of morality you can't be held responsible for the deaths of five people in a Trolley Problem-esque scenario but the people will still be dead and consequentialism is the view that consequences trump all other considerations, like adherence to a deontological moral code, acting as a virtuous person would act, acting in a way that is perfecting of one's teleological nature etc.
Whether or not to hold people responsible for certain actions, like everything else on consequentialism, is, for us, a matter of determining whether that would lead to the best consequences.
Now, having said that, in practice consequentialists will behave like deontologists, virtue ethicists etc. quite a lot of the time. The reason for this is that having everyone go around making individual act utilitarian calculations for every potentially moral decision would likely be catastrophic and consequentialists are committed to avoiding terrible consequences whenever possible. It is usually better to have rules that are held to be exceptionless (but which can be changed to some degree, as is exactly the case with law) and teach people to have certain virtuous qualities so that they won't be constantly looking for loopholes in the rules and so on.
Does this mean that deontology, virtue ethics etc. are correct after all? No! Because it's still all being justified on consequentialist grounds, which is how we decide which rules to have and what counts as a virtue etc. in the first place. They will be the rules and virtues that lead to the best real world consequences. Because the kind of philosophical scenarios under which it is morally correct to push a fat man off a bridge are carefully constructed to be as inconvenient as possible the rules and virtues that we use in practice will probably forbid pushing people off bridges or raising the kind of person who would do that. This will mean that once in a blue moon someone really will find themselves in such a scenario and that person will likely make the wrong decision. It will also mean that the majority of the time, people will be making the right decisions and the consequences, overall, will be better than they would be if we let people decide for themselves on a case-by-case basis whether murdering someone maximised utility or not.
Consequentialism will still have counter-intuitive results. It should! There's no reason to think that our intuitions are an infallible guide to what's right. However, the kind of consequentialism attacked in a lot of philosophical arguments is a pretty naive version and it would be much more productive for everyone if we focussed on the stronger versions.
Because it's still all being justified on consequentialist grounds, which is how we decide which rules to have and what counts as a virtue etc. in the first place. They will be the rules and virtues that lead to the best real world consequences.
The problem here is that what you need to justify is why you call some consequences better than others, because I might beg to differ. If you say "I just do" I would have to pull out my gun and say "well, I don't". In this scenario morality is reduced to might makes right, but then why call it...
Preface
I have trouble expressing myself in such a way that my ideas come out even remotely like they sound in my head. So please apply the principle of charity and try to read how you think I thought of it.
Tit for Tat
Tit for Tat is usually presented in a game between two players where each chooses to either cooperate or defect. The real world game however differs in two important ways.
First, it's not a two player game. We make choices not only on our single instance of interaction but also on observed interactions between other players. Thus the Advanced Tit For Tat not only defects if the other player defected against itself but also if it could observe the other player defecting against any other player that employs a similar enough algorithm.
Second, there is a middle ground between cooperating and defecting, you could stay neutral. Thus you can harm your opponent, help him or do neither. The question of the best strategy in this real life prisoners dilemma is probably still unanswered. If I see my opponent defecting against some of my peers and cooperating with others, what do I choose?
Agency
The reason why there even is a game is because we can deliberate on our action and can take abstract thoughts into account that do not directly pertain to the current situation, which I think is the distinguishing factor of higher animals from lower. This ability is called agency. In order to be an agent a subject must be able to perceive the situation, have a set of possible actions, model the outcomes of these actions, value the outcomes, and then act accordingly.
We could act in such a way that infringes on these abilities in others. If we limit their ability to perceive or model the situation we call this fraud, if we limit their set of possible actions or their ability to choose between them, we call it coercion, if we infringe on their ability to value an outcome, we call it advertising.
Ethics
I propose that the purpose of our moral or ethical intuitions (I use the two words interchangeably, if there is a distinction please let me know) is to tell us whether some player defected, cooperated or stayed neutral, and to tell us who we should consider as having a close enough decision algorithm to ourselves to 'punish' third players for defecting against them. And I further propose that infringing on someones agency is what we consider as defecting.
Value Ethics
Utilitarians tend to see defecting or cooperating as pertaining to creation or destruction of values. (Edit:) Three things bother me about value ethics:
1. Valuations between different people can't really be compared. If we shut up and multiply, we value the lives of everybody exactly the same no matter how they themselves value their own life. If there are chores to be done and one person claims to "not mind too much" while the other claims to "hate it with a passion" we can't tell if the emotional effect on them is really any different or maybe even the other way round.
2. It makes you torture someone to avoid an insanely huge number of dust specs.
3. It makes you push a fat man to his death.
Agency ethics
Instead I propose that defecting in the real world game is all about infringing on someone's agency. Thus we intuit bankers who destroy an insane amount of wealth while not as good people still as neutral because they do not infringe on agency. At least that is my moral intuition.
So infringing on agency would make you a bad person, while not infringing on agency doesn't make you a good person. What makes you a good person is increasing value. Maybe agency is more fundamental and you cannot be a good person if you are a bad person, but maybe you can be both. That would create cognitive dissonance in people who consider ethics to be a singular thing and don't see the distinction, and that might be at the root of some ethics discussions.
Evil
In my version of ethics it counts as evil to push the fat man or to switch the tracks, as that would mean deliberately causing a death of someone who doesn't want to die. I would let the five die and not feel guilty about it, because I am not the cause of their deaths. I make a fundamental distinction between acting and not acting. If I hadn't been there the five would still die, so how could I be responsible for their deaths? I am aware that this view makes me evil in the eye of utilitarians. But I see less people acting consistent with utilitarianism than I see people arguing that way. Then again, this perception is probably heavily biased.
Conclusion
I don't really have a conclusion except of noticing that there exists a disagreement in fundamental morality and to inform you that there exists at least one person who considers infringing on someone's agency as defecting in a prisoner's dilemma.