Eliezer, I think there's a human equivalent to your AI local-rationality repair mechanism: "cognitive dissonance".

This sounds like an error correcting code. There will be a limit to how much noise it can repair.