One additional unlikely event happening, or even one event being a foregone conclusion rather than a coin flip, will wipe out massive gains from model improvements, sometimes across thousands of predicted events.
Note: Assuming we are talking about binary predictions
It is true that one unlikely event happening can have arbitrarily high cost, but in practice people only bet up to the ~99% confidence level, so at most they incur log(0.01)=-4.6 nats of penalty.
It is not true that a foregone conclusion that was predicted as a coin flip can cause massive g... (read more)
Note: Assuming we are talking about binary predictions
It is true that one unlikely event happening can have arbitrarily high cost, but in practice people only bet up to the ~99% confidence level, so at most they incur
log(0.01)=-4.6nats of penalty.It is not true that a foregone conclusion that was predicted as a coin flip can cause massive g... (read more)