You are missing at least two options.

First, our knowledge of physics is far from complete, and there can be some reasons that make interstellar colonization just impossible.

Second, consider this: our building technology is vastly better than it was few thousands years ago, and our economic capabilities are much greater. Yet, noone among last century rulers was buried in tomb comparable to Egyptian pyramids. The common reply is that it takes only one expansionist civilization to take over the universe. But number of civilizations is finite, and colonization can be so unattractive that number of expansionists is zero.