I do not know of any case where someone has said that they "should have known better" after making a false positive, say, "I knew I shouldn't have used the seat belt on the buss, we did not crash after all".

Possibly the phrase "I needn't have bothered [..] after all"? e.g.:

Your place is delightfully homely and very tastefully decorated with a kitchen that was so well equipped with quality cooking implements that I needn’t have bothered bringing my own! - http://www.narooma-mystery-bay.com.au/Comments.htm

It's not directly saying they should have known, but it is saying they judged so inaccurately that reality took them by surprise so much it was worth commenting on.

Also the phrase "I don't know what we were worried about" fits a similar template - not a scolding for not knowing, like "should have known" is, but yes an admission of feeling overprepared for something which didn't happen and now questioning the reasons they had earlier.