However, if this next roll comes up 20, we are witnessing an extremely improbable sequence, so improbable that I have to start considering that the die is loaded."

The equivalent of "considering that the die is loaded" in your example is "the previous owners did a bad job of maintaining the house". It's indeed makes sense to come to that conclusion. That's also basically what your wife did.

Apart from that the difference between sequences picked by humans to look random and real random data is that real random data more frequently contains such improbable sequences.