I think you missed my point.

We know the Potters did not have their cloak because Dumbledore said so in his note to Harry.

To defend my parenthesis: earlier in the war, Voldemort taught Dumbledore that a human life is not of infinite worth. A corollary of this is that three humans lives may not be worth more than a deathly hallow. I.e. that the protection adequate for the safeguarding of three humans is not adequate for the protection of a deathly hallow (if the risk is that the deathly hallow falls into the hands of Voldemort).