If there are considerations that allow comparing alternative desires, and ways of influencing which ones you get, it may be a mistake to keep suboptimal desires. "They are just desires" stops the potentially fruitful inquiry. Desires are part of the world, subject to optimization.

A stronger position might be to say that it's usually inefficient to work on improving desires as opposed to working on something else. This probably depends on how broken they are in a particular case.