If Quirrell is Voldemort, which is more or less a foregone conclusion at this point, then James and Lily are "those who have thrice defied him". It seems natural to assume that he would know a certain amount about them. He also observed James's behaviour in the face of mortal danger to him and his family, which I imagine tells you a lot about a person.