You went straight for a baby right off the bat?

Yes. Of course. It was the blatant flaw in your description of morality. Unless one addresses that one first, there's hardly a need to discuss subtleties.

A baby is an actor which is specifically not free to do as it wishes, to do as it wishes, for a large range of very sensible reasons

If you're creating exceptions to your definition of morality for "sensible reasons", you should hopefully also understand that these sensible reasons won't be automatically understood by an AI unless they're actually programmed in. Woe unto us if we think the AI will just automatically understand that when we say "letting individuals do what they will" we mean "individuals with the exception of babies and other people of mentally limited capacity, in whose case different rules apply, mostly having to do with preserving them from harm rather that letting them do whatever".

In short your description of what morality entails isn't sufficient, isn't complete, because it relies on those unspoken and undescribed "sensible reasons". Once the insufficiency was shown to you you were forced to enhance your description of morality with ideas like "full agency" and the "capacity to rationally determine what will cause them harm".

And then you conceded of course that it's not just babies, but other people with mentally limited mental capacity also fall in that category. Your description of morality seems more and more insufficient to explain what we actually mean by morality.

So, do you want to try to redescribe "morality" to include all these details explicitly, instead of just going "except in cases where common sense applies"?