Well, it's hard to articulate. There's of course nothing wrong with assumptions per se, since axioms indeed are assumptions, my peeve is with the baggage that comes with it. People say things like "what if the assumptions are wrong?", or "I don't think that axiom is clearly true", or "In the end you can't prove that your axioms are true".

These questions would be legitimate if the goal were physical truth, or a self-justifying absolute system of knowledge or whatever, but in the context of mathematics, we're not so interested in the content of the assumptions as we are in the structure we can get out of them.

In my experience, this kind of thing happens most often when philosophically inclined people talk about things like the Peano axioms, where it's possible to think we're discussing some ideal entity that exists independently of thought, and disappears when people are exposed to, say, the vector space axioms, or some set of axioms of set theory, where it becomes clear that axioms aren't descriptions but definitions.

Actually, you can ignore everything I've said above, I've figured out precisely what I have a problem with. It's the popular conception of axioms as descriptive rather than prescriptive. Which, I suppose OP was also talking about when they mentioned building blocks as opposed to assumptions.