Now what about the other parts? Are there any other reasons to either speak up or not speak up, besides avoiding someone's critique?

I try to helplessly flail in the general direction of that empirical cluster in another post, but in general I would advise that if you value other people's long term emotional well-being, my best guess says you really should use a strategy of speaking up rather than not, wherever not disproportionately dangerous.