Does it matter whether there is hope or not in the stories? We are less hopeful about this precisely because things haven't been nearly as impressive in the last fifty years as we thought they would be. A story set fifty years in the future with big Mars and Moon colones won't look hopeful, it will look at best overly optimistic, and possibly naive.
Moreover, this may actually be a signal that the genre has reached real maturity, that it isn't just gee-whiz look at that stories.
The claim that the genre has collapsed simply isn't born out when one looks at how popular Greg Egan, Charlie Stross, Alastair Reynolds, and Cory Doctorow are among others. The genre is in fact much more successful now. Fifty years ago, book shops didn't have separate sections for science fiction. Now they have large sections. Science fiction also is one of the most popular genres judging by the sales of used books. Science fiction is also far more accepted as a literary genre by academia, to the point where there's a separate area of study devoted to it. I have to wonder how much of what you are seeing is just the decline in average quality that occurs when a small thing becomes popular.
Just because the genre conventions have changed from your preferred form doesn't mean that the genre has collapsed. This sounds a bit akin to fans who when something in their favorite franchise changes become convinced that it has been Ruined Forever. (Not linking to TVTropes for obvious reasons).
Fifty years ago, book shops didn't have separate sections for science fiction. Now they have large sections.
I can't speak to fifty years ago, but I can tell you that 47 years ago Bookland, a smallish and not extremely ambitious bookstore in a suburb of Wilmington, Delaware definitely did have a science fiction section.
SIAI benefactor and VC Peter Thiel has an excellent article at National Review about the stagnating progress of science and technology, which he attributes to poorly-grounded political opposition, widespread scientific illiteracy, and overspecialized, insular scientific fields. He warns that this stagnation will undermine the growth that past policies have relied on.
Noteworthy excerpts (bold added by me):
In relation to concerns expressed here about evaluating scientific field soundness:
Grave indictors:
HT: MarginalRevolution