I guess I'm too used to fanfics that pick holes in the story in what I perceive as an uncharitable manner (my thoughts usually go something like "in the LCPW, black-box wards that detect and record all magic performed in a house are common knowledge"). But upon reflection, you're right - if there were a reasonable explanation, Harry's failing to be confused wouldn't have been treated as such a big deal in the narration.