That's a very interesting question and it is unfortunate that it did not get more traction, because I think I could learn a lot by reading more answers. In no way what follows is a definitive answer, it is just my own take.
First, how can we settle who has been a better forecaster so far?
A naive answer would be, let's pick the forecaster who has the lowest cross-entropy, i.e. the same way when we train a binary classifier which outputs probabilities, we pick the model which minimises the cross-entropy. I say this answer is naive because if we ta... (read more)
That's a very interesting question and it is unfortunate that it did not get more traction, because I think I could learn a lot by reading more answers. In no way what follows is a definitive answer, it is just my own take.
A naive answer would be, let's pick the forecaster who has the lowest cross-entropy, i.e. the same way when we train a binary classifier which outputs probabilities, we pick the model which minimises the cross-entropy. I say this answer is naive because if we ta... (read more)