Eliezer, your observers would hopefully have noticed hundreds of millions of years of increasing world-modeling cognitive capability, eventually leading to a species with sufficient capacity to form a substrate for memetic progress, followed by a hundred thousand years and a hundred billion individual lives leading up to now.

Looking at a trilobyte, the conclusion would not be that such future development is "impossible", but perhaps "unlikely to occur while I'm eating lunch today".