Usually I try to give similar answers that halt the line of conversation.

"I've never cared for sports, I shouldn't play for health reasons, it's not interesting to me, I don't understand the point, I've got other things to do, my dog was killed by a rogue football and I've never been the same since that fateful day", etc.

I've never actually answered "No" to the question "Why not?!", but I feel as though I should try, now...

So, I've never really let it progress beyond that point. As a kid, I did that with both religion and politics, by giving noncommittal answers.