Not to mention this, even before the troll:

Harry had politely declined tea, even knowing that Professor Quirrell would know what it meant. He’d considered bringing his own can of soda—but had decided against that as well, after realizing how easy it would be for the Defense Professor to teleport in a bit of potion, even if the two of them couldn’t touch each other with direct magic.

I thought that he just didn't want to talk to Quirrell at the time and wanted to continue his walk alone for another hour.