Part of being an effective communicator is optimizing what you say for your audience. You shouldn't take pride in not trying to do this. Train your brain to make optimal use of jargon given your audience, not to minimize your use of jargon.

New college professors often have trouble teaching "down" to the level of their students, but the solution for them is not to lower the complexity of their conversations with everyone, but rather to train their brains to respond differently when talking to students as opposed to colleagues.