So why does it make sense to demand that the human (who, in the chinese room is really playing the same role as physics) understand Chinese in order to believe that the room-human system does?

It doesn't.

I think the argument obscures what might be a genuine point, which I look at here: http://lesswrong.com/lw/lxi/hedoniums_semantic_problem/