I think - though I'm not certain - that you are right that in all but the most marginal seats you'd never find a politically unafiliated / passionate about a political issue group large enough to swing a seat. I don't think that implies that you shouldn't try to game democracy though - there are certain known flaws in the democratic system we have which exist (and swing elections) independently of whether people knowingly exploit them or not.