The line about having been sorted into Ravenclaw could be as fake as the Fuyuki City thing, Scrimgeour's play. Quirrell's apparent failure could just be a way of getting temporarily detained, while Dumbedore's looking for Riddle and Harry wants his help. His cover could actually be pretty solid, so he'd just shrug off Scrimgeour's suspicions once it's time to go.