Yes. Not even being attracted to anyone, physical contact (especially the close kind in grappling) tends to have me consciously trying not to panic over those sorts of implications. It might be a cultural thing? Possibly mixed with personality type/experience?

(Yes, it is annoying.)