In my limited experience with children i have found that children will only do this when they are being ignored. I have only experienced it once myself and you quickly see the child is teasing you by their eyes and uneasy movement. The benchmark I use is repeating the same question again at a later time, that means they either did not understand or are teasing.

Any thoughts about how to tell whether a child wants real answers? If it is for the social interaction, what's a graceful way to handle it without squelching real curiosity?

My intuition is that figuring out whether the child wants a "real answer" is not a good use of your time. Instead, you should treat every question as a real question and try to come up with a rule about when to answer them. Reasons to stop answering might include (1) your lack of interest in answering, (2) the answer is beyond the child's ability to comprehend - I can easily image a conversation with a 4 year old that basically resolves to "Go learn calculus," which isn't a useful answer.

And if you decide to stop answering, you can just tell the child that you don't want to answer more questions. If the response to that is "why," then it's pretty safe to say that falsifiable statements are not where the conversation is located.

Sometimes with small children, I get the impression they're asking "why?" for the social interaction rather than to actually get answers.

The story I tell myself about this is that at some point every child learns that there is a magic word, "why", that always keeps a conversation going. I will test this hypothesis the next time a kid does this to me by responding with nonsense.

"Why?"

"Because zebra donkey tomatillos."

"Why?"

"See, you're not even listening."

"Why?"