I think it's worthwhile, at this point, to define exactly what one means by 'sexist'. According to Wiktionary, it means (as a noun):

A person who discriminates on grounds of sex; someone who practises sexism.

And as an adjective:

Unfairly discriminatory against one sex in favour of the other.

Now, looking at 'discriminate', we have two definitions:

1. (intransitive) To make distinctions.

And:

1. (intransitive, construed with against) To make decisions based on prejudice.

This definition leaves it a little unclear as to whether describing any differences between genders (including the obvious ones) is sexist, or whether only describing discriminatory differences between genders is sexist. Personally, I had always assumed that any difference counted as sexist, but only discriminatory differences counted as bad things to say; that is, that saying that boys are on average stronger than girls is both true and sexist.

From your post, I think that you are using a different definition of 'sexist', where only the discriminatory uses are considered valid examples.