Brain size sets an upper bound on intelligence, but it doesn't determine it. There are plenty of species with brains much larger than ours that are nevertheless very stupid, and there's little or no correlation between brain size and intelligence among humans. Since the Boskops appear to have not gone anywhere and not left behind any signs of writing or technology, the logical conclusion is that they weren't smart enough to have either.