I think that you must differ from the average person in some way that makes it not enjoyable for you. Perhaps you are more sensitive to certain sounds and find them unpleasant. Perhaps you weren't exposed to music at a very young age. Your brain might be "wired" differently than average.