I think it's quite unlikely, yes.

It seems like a natural class of explanations for the fermi paradox, one which I am always surprised never gets more people coming up with it. Most people pile into 'intelligent systems almost never appear' or 'intelligent systems have extremely short lifespans'. Why not 'intelligent systems find it vanishingly difficult to spread beyond small islands'? It seems more reasonable to me than either of the two previous ones, as it is something that we haven't seen intelligent systems do yet (we are an example of one both arising and sticking around for a long time).

If I must point out more justification than that, I would immediately go with:

1 - All but one of our ships BUILT for space travel that have gone on to escape velocity have failed after a few decades and less than 100 AUs. Space is a hard place to survive in.

2 - All self-replicating systems on earth live in a veritable bath of materials and energy they can draw on; a long-haul space ship has to either use literally astronomical energy at the source and destination to change velocity, or 'live' off only catabolizing itself in an incredibly hostile environment for millennia at least while containing everything it needs to set up self-replication in a completely alien environment.

Edit: a friend of mine has brought my attention to this paper:

http://www.geoffreylandis.com/percolation.htp

it proposes a percolation model of interstellar travel in which there is a maximum possible colonization distance and a probability of any successful colonization spawning colonizers themselves. It avoids all three above-posited explanations for the fermi paradox and instead proposes a model of expansion that does not lead to exponential consumption of everything.