Can we get in some agreed upon middle ground?

A simple daily-iterated formula to start: WEIGHT = WEIGHT - WEIGHTBURN + FOOD

My assumptions are that WEIGHT is the person's current weight. WEIGHTBURN is the amount the person burn per every day from energy consumption + bodily maintenance. FOOD varies from person to person.

My questions for you:

But it is possible that some of the "calories in (the mouth)" may pass through the digestive system undigested and later excreted? Could people differ in this aspect, perhaps because of their gut flora?

Not unreasonable. I remember reading that while brocoli has more calcium than milk, the composition of milk allows the calcium to be absorbed better. In fact, the components of brocoli seem to contain something that actually inhibits calcium absorption!

More generally, I assume your reasoning here to be that actual food digestion is not a 1:1 to, say, food labels. Correct? (I assume that food labels use some sort of average, say, 10,000/100 = x per 100g. Correct me if this is wrong please!)

Also, what if some people burn the stored fat in ways we would not intuitively recognize as work? For example, what if some people simply dress less warmly, and spend more calories heating up their bodies? Are there other such non-work ways of spending calories?

Define your 'work'. Is it physical activity without any body maintenance? Keeping your body temperature, for example. Digesting food also takes 'work'. I don't think you can burn so much calories from exercise alone, in fact. Calorie counting is a better choice for fat loss than walking/running distance.