"Vegetarianism isn't really rational in the first place"
You're throwing around "rational" in an unhelpful way. The question at issue is: does going vegetarian make the world better? There's a good argument that the number of animals living on factory farms is proportional to the number of meat eaters, so if you value the suffering of animals then decreasing the number of people who eat meat should be helpful.
"probably having a net negative impact by encouraging terrorist groups like PETA."
PETA may sometimes get classified as terrorists but they're clearly not what most people will think of when they hear "terrorist"; they don't go around blowing things up. This is the noncentral fallacy. While I think PETA isn't going about things well, if you value the suffering of animals I have a hard time seeing how "maybe encouraging PETA a bit" can outweigh many fewer animals living on factory farms.
PETA is without question a terrorist organization. They act as a front for recruitment for terrorist groups such as ALF, they give money to such groups, they support their activities, they send out large amounts of propaganda, they have significant overlap in membership with said terrorist groups and put terrorists in leadership roles... the list goes on. They DO, in fact, go around burning stuff down, commit arson, and I know on at least one of those occaisions left their dog locked up in their truck on a hot day while they were out "liberating' rabb...
I notice that a large fraction of Effective Altruism people are vegetarian. This makes sense: in general Effective Altruists take moral issues seriously, even when that means changing your lifestyle. I'm not sure it's a good balance, though.
One way to think about this is to convert it into money. How much would I need to be paid to give up eating meat? All animal products? How much money would I need to spend on myself to be about as happy as I would be with less money but continuing to eat animals? I'd probably be willing to go vegetarian for about $500/year, vegan for maybe $2000/year.
It turns out you can probably pay to convince other people to be vegetarian much more cheaply than that. I estimate the cost of a vegetarian-year at $4.29 to $536 while Brian Tomasik estimates $11 with better methodology (which I looked at). This is by placing ads on facebook for a site where people can watch an animial cruelty video and ideally become vegetarian or vegan.
If you would get more than $11/year worth of enjoyment out of continuing to eat meat, why not give $11/year to convince someone else to not eat meat for you? Or give $50/year and be on the safe side?
(While you're giving money, you should probably give it to the organization that you think will do the most good with it, which I think is probably one of GiveWell's top charities. The nice thing about money as opposed to actions is that it's easy to redirect.)
I also posted this on my blog.