If I won't hire black people because, on average (I believe) they are not quite as smart as white people, wouldn't I be better off just not hiring the individuals who seem dull to me?

The problem is that someone else will notice that you've hired more white people than black people and accuse you of racism and at least in the US the law will back them up.