(...) and we have no reason to believe that our current observations apply to the outside.

Not quite true.

Occam's Razor says things are probably the same everywhere, whether inside or outside, which points towards our observations applying to the outside.

Since our observations apply to pretty much all of the inside, we should infer that they are also more likely to apply to pretty much any part of the outside because inference usually works with large samples (and the entire observable universe is something I'd consider a fairly large sample, if nothing within the observable universe goes against our current observations).

So we have two principles we could use to believe that our current observations apply to the outside. This requires that our priors for inference and Occam's Razor working be set pretty high, obviously.

Even if the inflation stops or slows down, that's still what "cosmological horizon" means.

This merely implies that if the inflation does stop or slow down, the cosmological horizon will also happen to coincide with "how far a signal could possibly travel until the complete death of the universe / the end of time", for c-bound velocities. If there's an upper bound on the spatial size of the universe, then the cosmological horizon might not exist in reality at this point. If there isn't but it turns out we will never reach maximum entropy, then the cosmological horizon also doesn't exist here.

Fun stuff to think about, IMO.