If different races have different skin, muscle/bone structure, genetics, and maybe other things, shouldn't it follow that different races could have different brains, too?

They do. Even if you don't want to go into IQ measurements, different races have different brain volume just for starters. See e.g. Cochrane:

...average brain size is not the same in all human populations. Average cranial capacity in Europeans is about 1362; 1380 in Asians, 1276 in Africans. It’s about 1270 in New Guinea. Generally there is a trend with latitude – brain volume is lowest near the equator. And no, despite Gould’s bushwa, there is nothing especially difficult about measuring brain volume. Direct measurement of a healthy brain is best; but that is now done, using magnetic resonance imagery, and the results are about the same – a mean black-white difference of about 1 standard deviation.