Yes, rationality could use more competitive tests of skill. I'm not sure about having them take a year, though. Have you thought about ways to make it faster?
A simple fix would be to include predictions about particular parts of the year. For example, have a bunch of predictions about each quarter, on top of the ones about the whole year. And then you could have an extra prize for each quarter where you score only that subset of predictions.
You could easily go more short term, like "what are your predictions for September", but I think this requires more participation and work from everyone so maybe it would be better as a second step that you do only if the relatively relaxed yearly tournament has turned out to be cool.
So, I've been thinking about prediction markets and why they aren't really catching on as much as I think they should.
My suspicion is that (beside Robin Hanson's signaling explanation, and the amount of work it takes to get to the large numbers of predictors where the quality of results becomes interesting) the basic problem of prediction markets is that they look and feel like gambling. Or at best like the stock market, which for the vast majority of people is no less distasteful.
Only a small minority of people are neither disgusted by nor terrified of gambling. Prediction markets right now are restricted to this small minority.
Poker used to have the same problem.
But over the last few decades Poker players have established that Poker is (also) a sport. They kept repeating that winning isn't purely a matter of luck, they acquired the various trappings of tournaments and leagues, they developed a culture of admiration for the most skillful players that pays in prestige rather than only money and makes it customary for everyone involved to show their names and faces. For Poker, this has worked really well. There are much more Poker players, more really smart people are deciding to get into Poker and I assume the art of game probably improved as well.
So we should consider re-framing prediction the same way.
The calibration game already does this to a degree, but sport needs competition, so results need to be comparable, so everyone needs to make predictions on the same events. You'd need something like standard cards of events that players place their predictions on.
Here's a fantasy of what it could look like.