Agreed, except there's no particular reason for a Dark Lord to actually leave a survivor when he can just have his minions disseminate it. (Or do so himself as Quirrell; we have no knowledge of how long this story has been around.)

ETA: Actually I should say my first thought was that Voldemort destroyed the dojo not out of anger, but simply to make sure that no rival wizard ever got the awesome martial arts training that he did. This seems strongly implied when he says: "You are wondering where this marvelous dojo is, and whether you can study there. You cannot."