Harry thinks that Voldemort wouldn't be much of a challenge, but Quirrell would be another matter entirely.

Doubtless Harry based that belief on his extensive experience with Voldemort. Dumbledore's opinion, on the other hand:

"Aye, it is he," Albus said. "Azkaban has endured impenetrable for ages, only to fall to an ordinary Animagus potion. It is too clever and too impossible, which was ever Voldemort's signature since the days he was known as Tom Riddle. Anyone who wished to forge that signature must needs be as cunning as Voldemort himself to do so. And there is no one else in the world who would accidentally overestimate my wit, and leave me a message I cannot understand at all."