Let's put it this way, the two reasons you've given for not voting are:

1) You're unlikely to affect the outcome anyway.

2) If enough people don't vote the government will have less legitimacy and this can have positive effects.

Since the logic of these two reasons contradict, would you mind telling me which is your true rejection?

I'm another non-voter, largely (or medium-largely) for the reasons Konkvistador gives. But it's not the legitimacy of government that I wish to weaken. Places where government, even bad government, is not taken seriously are not nice places to live. If there's an institution or a cultural value that I wish to see weakened it's the people's romance.

In general I see nothing inconsistent about a democracy where most people voluntarily abstain from voting. A norm of not voting would require low amounts of sectarian conflict and large amounts of social trust, which don't exist in very many democracies. But as goals go I think low levels of sectarianism and high levels of social trust are superior to (and at cross-purposes with) high levels of voting.