Also, the note from Dumbledore specifically says that James left the cloak in Dumbledore's possession before he died. (Maybe to prevent it from getting into Voldemort's hands.)

Since Dumbledore was the strongest magician of them, giving the cloak to Dumbledore to fight more effectively might have been effective.