You should note which domains make you surprised frequently, and consider explicitly revisiting your models and assumptions in that domain.
You should notice which domains don't have great models, and check that you are surprised there as frequently as you'd expect - or note that you fail to be surprised, or that you treat everything as equally likely in those domains, and consider if you're doing well.
I read the article found at:
http://www.fastcompany.com/3035671/how-i-get-it-done/a-new-technique-for-creating-more-aha-moments-the-surprise-journal
I started writing a few things down. I realize that the point of the surprise journal is to give your brain a little reward for noticing something. So, the actual keeping of the journal is surplus. You could write it down and then throw it away.
But that feels intuitively like an open loop. Is anyone doing anything with a surprise journal after writing it down?