People say things like "what if the assumptions are wrong?"

That's a valid question in a slightly different formulation: "what if we pick a different set of assumptions?"

"In the end you can't prove that your axioms are true"

But that, on the other hand, is pretty stupid.

It's the popular conception of axioms as descriptive rather than prescriptive.

Well, normally you want your axioms to be descriptive. If you're interested in reality, you would really prefer your assumptions/axioms to match reality in some useful way.

I'll grant that math is not particularly interested in reality and so tends to go off on exploratory expeditions where reality is seen as irrelevant. Usually it turns out to be true, but sometimes the mathematicians find a new (and useful) way of looking at reality and so the expedition does loop back to the real.

But that's a peculiarity of math. Outside of that (as well as some other things like philosophy and literary criticism :-D) I will argue that you do want axioms to be descriptive.