Does it matter whether there is hope or not in the stories? We are less hopeful about this precisely because things haven't been nearly as impressive in the last fifty years as we thought they would be. A story set fifty years in the future with big Mars and Moon colones won't look hopeful, it will look at best overly optimistic, and possibly naive.

Moreover, this may actually be a signal that the genre has reached real maturity, that it isn't just gee-whiz look at that stories.

The claim that the genre has collapsed simply isn't born out when one looks at how popular Greg Egan, Charlie Stross, Alastair Reynolds, and Cory Doctorow are among others. The genre is in fact much more successful now. Fifty years ago, book shops didn't have separate sections for science fiction. Now they have large sections. Science fiction also is one of the most popular genres judging by the sales of used books. Science fiction is also far more accepted as a literary genre by academia, to the point where there's a separate area of study devoted to it. I have to wonder how much of what you are seeing is just the decline in average quality that occurs when a small thing becomes popular.

Just because the genre conventions have changed from your preferred form doesn't mean that the genre has collapsed. This sounds a bit akin to fans who when something in their favorite franchise changes become convinced that it has been Ruined Forever. (Not linking to TVTropes for obvious reasons).