Did the speaker imply the second part? Maybe... but probably not.
They did not logically entail it but they did conversationally implicate it (see CGEL, p. 33 and following, for the difference). As per Grice's maxim of relation, people don't normally bring up irrelevant information.
B, interrupting: Didn't say it does!
At which point A would be justified in asking, “Why did you bring it up then?” And even if B had (tried to) explicitly cancel the pragmatic implicature (“It's wrong to stab your neighbor and take their stuff” -- ”I won't comment on that; on a totally unrelated note, your face is ugly”), A would still be justified in asking “Why did you change the topic?”
B here is violating Grice's maxims. That's the point. He's not following the cooperative principle. He's trying to insult A (perhaps because he is frustrated with the conversation). So applying Gricean reasoning to deduce B's intended meaning is incorrect.
If A asks "why are you changing the subject?", B's answer would likely be something along the lines of "And your mother's face is ugly too!".
Another month has passed and here is a new rationality quotes thread. The usual rules are: