But people don't claim that their inside jokes are the highest form of culture and that others are somehow deficient for not wanting to join in on it.

I understand that if you invest some effort E into appreciating something, you'll appreciate it. The fact that I appreciate it for some (potentially huge) E does not somehow justify the effort -- you can say that about anything.

The appropriate comparison would be "what ways of amusing myself for that level of personal investment are the best"? And given these opportunity cost considerations, it's quite understandable why the utter indifference of the public is a strike against the field.