I don't at all disagree with your point, "none of your business" is just defensive language that I find that people use when they don't want to justify something that they have no justification for.

It is something people use when people ask for information that they do not have a right to have, asked for justifications that the recipient should not be expected to give or give advice or make demands that they do not have the power or right to make. In many cases responding with a justification instead of "none of your business" or an equivalent rejection is a social mistake that just encourages more dominance displays and boundaryless behavior on the part of whoever is making the presumption.