Thank you, Vladimir, wedrifid, Cayenne.

Now, how would an exercise to train this 5-second-skill look like?

Read out to a group questions of the form 'why X?' where X itself is a controversial statement for which arguments for and against can be found. This shall encourage them to think of whether X is true itself. X could be very probable, something like 'rationality is the best way of life', or something improbable. This way, the group shall learn to avoid the urge to rationalize while at the same time trying to avoid the opposite, namely feel the urge to crush every statement.

Could this work? How could one modify it?