They don't necessarily have to be--but then it should be easy to identify the radicals, as they go off to start their own groups.

If there is another group close to the moderates, say another political party, they may go join that one since that would be more straightforward than waiting for the extremists (who also may have more affective attachment to the label) to leave.