As long as you have an existing set of questions with known answers that are unknown to participants of the game you can have instead feedback.
Public knowledge that you can find on Wikidata works if you have an offline tournament. For an online tournament, it can use data from nonpublic experiments. The CASP tournament for protein structure prediction uses that method. For our purposes, I think surveys make good experimental data.
But in that case, it isn't really about prediction anymore. A game like that rewards knowledge, not the ability to do research and deal with probabilistic information.
Someone who has read a lot of Wikipedia, or who happens to have read papers on topics similar to the experiment in question, could outperform someone who constructs predictions very rationally but from a different set of domain knowledge facts. This makes it closer to a quiz show, i.e. a less original and less interesting event.
A slow, online tournament (where everyone has the same internet to do research in) greatly reduces the value of blunt knowledge and makes success more dependent on the ability to weigh evidence.
So, I've been thinking about prediction markets and why they aren't really catching on as much as I think they should.
My suspicion is that (beside Robin Hanson's signaling explanation, and the amount of work it takes to get to the large numbers of predictors where the quality of results becomes interesting) the basic problem of prediction markets is that they look and feel like gambling. Or at best like the stock market, which for the vast majority of people is no less distasteful.
Only a small minority of people are neither disgusted by nor terrified of gambling. Prediction markets right now are restricted to this small minority.
Poker used to have the same problem.
But over the last few decades Poker players have established that Poker is (also) a sport. They kept repeating that winning isn't purely a matter of luck, they acquired the various trappings of tournaments and leagues, they developed a culture of admiration for the most skillful players that pays in prestige rather than only money and makes it customary for everyone involved to show their names and faces. For Poker, this has worked really well. There are much more Poker players, more really smart people are deciding to get into Poker and I assume the art of game probably improved as well.
So we should consider re-framing prediction the same way.
The calibration game already does this to a degree, but sport needs competition, so results need to be comparable, so everyone needs to make predictions on the same events. You'd need something like standard cards of events that players place their predictions on.
Here's a fantasy of what it could look like.