So this is a case where I anticipated that my values were likely to drift, in a way incompatible with my current values, and removed myself from a situation I thought likely to cause that drift. (Mind you there were plenty of other reasons.)

Not to say that you weren't right to do that, but I notice that the religious will sometimes avoid consorting with the non-religious for the same reason.

Faced with an experience that one foresees being changed by, how should one decide whether to go ahead with it? Given several sets of values, such that from the standpoint of any of them all the others look worse, how to decide which to adopt?