Who authorizes messengers from God?

God obviously.

It's not like He has a public key,

There are actually quite a few rules given to determine if a messenger is from God. Jesus for instance said "If any man will do his will, he shall know of the doctrine, whether it be of God, or whether I speak of myself", then there is the qualification in Deuteronomy, the requirement in John, the experiment in Alma, the promise in Moroni, and some details in the D&C. It is somewhat of a bootstrapping problem as one must already trust one of those sources, or the person presenting those sources, enough to move forward in trying to verify the source and messenger.