it seems pretty hard to construct anything like an observer without probabilities, so there are only observers in as much as one is looking at the world according to the Born Rule view

Even better than that - there can be other ways of making observers. Ours happens to be one. It doesn't need to be the only one. We don't even need to stake the argument on that difficult problem being impossible.