Yeah, whether the lack of popularity of jazz and classical music is caused by their slowdown or their slowdown is caused by their lack of popularity is one of those tricky questions. There's definitely causation going both ways, but it's really hard to tell which part has more impact, or which one changed first to start off the decline (and if they're both driving each other now, does it even matter which one started things off?). If I had to guess, I'd say that the rise of rock, pop, and (slightly later) hip-hop music gave people new musical options, and this led to the decline in popularity of jazz and classical because, for social reasons, their new competitors were more appealing to most people. This would then cause talented musicians to be attracted to the newer genres, making the older ones worse and less popular, and kicking off that whole vicious cycle. I'm completely speculating here, though; music history is not my forte.

The example of a single band like AC/DC is interesting, and them running out of good music is quite plausible, but other plausible explanations exist too. It would make just as much sense to say that the band members and their relationships changed with time, and since most ways in which a band can exist do not lead to the highest quality music, these changes made them worse. This would explain their declining quality just as well as your hypothesis, and it's hard to tell which is true.

It is certainly true that each song is unique, and once made, that point in song-space will no longer be fresh for the rest of time. The question is whether or not the space of good songs is so large that we don't care about specific points; if it is, your concern is unfounded. I'm not really sure what the truth is here, but I'm leaning toward the space being quite large. Music has a lot of different knobs to turn to make songs original (even while staying within the same genre), and I think you'd need more convincing evidence than "these genres' good output is reducing along with their popularity" to conclude that we've made all the good stuff already.

A bit of supporting evidence for this: classical music can probably be reasonably considered to be a more constrained genre than jazz, with less options for songs because of its stricter style, and therefore a smaller total number of "substantial works". It has existed for hundreds of years longer than jazz music has, so it would be very odd for both genres to be running out of good ideas at the same time, because, based on its longevity, you'd expect jazz music to last much longer than a few decades. Granted, those few decades were in the 20th century only, when much more music was being created than in previous centuries, but the point still stands, if a bit weaker than it could have been. This seems to imply that it's not actually running out of ideas that's causing the decline in substantial new work, if I'm understanding it right. As to what it actually is, though, I don't feel qualified to say.