This kind of drift will presumably happen given enough time, but it's worth noting that (for obvious reasons) Esperantists tend to be more disciplined about maintaining the integrity of the language than is typical among speakers of most languages, and they've been pretty successful so far.

One advantage Esperanto has over natural language, is that nearly all of its speakers speak it as a second language. That is way most of its learners are self-consciously trying to maintain its integrity.