There was more to slavery than estimations of intelligence - the justifications varied wildly, were usually absurdly simple to disprove, and often contradicted each other ("they were designed by God to be enslaved by superior races" vs "they have weaker self-control and would kill/rape us if left unchecked", for example.)

However, the point that it was a failure of rationality, not ethics, is still valid. Unfortunately that was the OP's point as well.