Can you explicitly write down the definition of actually_should? There seems to be an implicit assumption in the post that while "the code that produces the should statements" is complex and distinct for each agent, "the code that uses them" is simple and universal. I am not sure how this is warranted.

The first claim I would like to make is that Actually_Should is a well-defined term - that AI_should and me_should and you_should are not all that there is.

I don't understand how the two parts of this sentence are related.