"there's no reason why the end result would be moral in any way. "

This is wrong. The reason is that everyone is doing what they want, which is, on average, more likely to benefit people than having to do what they do not want, since people typically want to do things that benefit them and avoid things that do not.

The above is, in fact, the basic but extremely simplified reason why no one yet has been able to come up with a better system.