I'm not even worried about cult issues - I'm fairly confident that I'd notice and extricate myself from such a situation if there was one. (I've done so before, more than once.) It's more of an issue of consistently being a certain kind of person: Rather like Alicorn and Eliezer don't lie, I don't base my actions on other peoples' assumption or pressures regarding what I should or will do. (It may be noteworthy that this isn't the same as not taking peoples' likely reactions to my actions into account.)