I guess that EY's tale here about holding open doors and letting people know they left the car trunk open is why I keep doing this. If I modify myself to completely ignore these little things... what will I lose? Can I really just not ever give anyone the time? What about all those times when they really did just need to know the time, or wanted to charge their phone, or whatever?

People don't leave their car trunks open for deception unless they're kidnappers. If you can't tell if people are lying or not, please just ignore them. Otherwise you're encouraging the dishonest ones to harass other people too.