Some helmets are more acceptable than others. I've had people say things like "You know you are inside now" to me when I take the elevator to my office and still have my bike helmet on. I do take it off when I get to my office. In contrast, hard hats are generally seen as reasonable to wear for certain people even when they are not on a construction site. The presumption is that someone wearing a hard hat does work in construction or something similar and just didn't want to take the hat off, I suppose.