Is there some reason to think that physiognomy really works?

It is the case that appearances encode lots of information, because lots of things are correlated. For example, height correlates with intelligence, probably because of generic health factors (like nutrition). Nearsightedness and intelligence are correlated, but whether this is due to different use of the eyes in childhood or engineering constraints with regards to the brain and the skull is not yet clear. The aspect ratio of the face correlates with uterine testosterone levels, which correlates with masculinity. Those are just the particularly famous ones; an expert could keep going for some time.

The question of whether or not people's naive impressions correspond to actual correlations, though, is less clear. Overall, it looks like stereotypes are more likely to be right than not, and it's more likely that the ground fact shifts the impression than the impression shifts the ground fact. (How is thinking that blockier faces are found on more masculine men going to adjust the response of bone growth to testosterone in utero?)