That's been bothering me ever since chapter 104 came out. It doesn't seem likely that Quirrel would have just missed that, so I see a couple possibilities:
1) Part of Quirrel's plan somehow requires Harry to use the time turner once, so by having Harry go back five hours he limits Harry's use of it as much as possible.
2) It's a trap. Leaving via time turner is an excellent escape mechanism, and Quirrel knows that Harry has a tendency to overuse it for that purpose. So if Harry does manage to do something unexpected, Quirrel's hoping he'll go for the ti...
Heh. From Chapter 17: