This is a valid point: to reverse a joke you can't just change the labels around, because jokes that apply to stereotypes only make sense when they apply to a stereotype that people know about. In your case you might try laughing at a joke about a mathematician who's so deep in thought that he walks into traffic.