Not particularly referring to your experience, but instead drawing from a few dozen rock festivals I've been to in the past decade.

You gain nothing from booing them, except possibly you signal...what? Being loud and opinionated? Being in a position of judgement and therefore high-status?

This is the main reason, for what I saw. Booing an act puts you on a higher level than the people who like it, and have therefore bad taste. In addition, it could also signal the membership to a different fan group.

Even assuming there's a signaling explanation, I cannot figure out the thought process that leads to booing. Like, they somehow get angry at the performers? Or is it morbid curiosity, and they wonder if it'll get even worse if the dancers get flustered?

A classic festival example that I personally witnessed. A few years ago I was at an heavy metal festival, with many groups performing the same day. There were a few extreme metal groups (Obituary, Slayer and Stormlord IIRC), and a very noisy group of extreme metal fans. Unfortunately, sandwiched between those acts, the organizers inserted Lacuna Coil (a roughly gothic metal group, much softer, with a wider fanbase outside of the metal community, including a fair share of teenage girls). Needless to say, the extreme metallers completely ruined the performance with boos, and at some point started launching plastic bottles. They were clearly trying to show which was the dominant group, and if you didn't boo, then you weren't part of such group.