I would like to put forth the argument that we already have multiple human-programmed "Strong AI" operating among us, they already exhibits clearly "intelligent", rational, self-modifying goal-seeking behavior, and we should systematically study these entities before engaging in any particularly detailed debates about "designing" AI with particular goals.
They're called "Bureaucracies".
Essentially, a modern bureaucracy - whether it is operating as the decision-making system for a capitalist corporation, a government, a non-profit charity, or a political party, is an artificial intelligence that uses human brains as its basic hardware and firmware, allowing it to "borrow" a lot of human computational algorithms to do its own processing.
The fact that bureaucratic decisions can be traced back to individual human decisions is irrelevant - even within a human or computer AI, a decision can theoretically be traced back to single neurons or subroutines - the fact is that bureaucracies have evolved to guide and exploit human decision-making towards their own ends, often to the detriment of the individual humans that comprise said bureaucracy.
Note that when I say "I would like to put forth the argument", I am at least partially admitting that I'm speaking from hunch, rather than already having a huge collection of empirical data to work from - part of the point of putting this forward is to acknowledge that I'm not yet very good at "avalanche of empirical evidence"-style argument. But I would *greatly* appreciate anyone who suspects that they might be able to demonstrate evidence for or against this idea, presenting said evidence so I can solidify my reasoning.
As a "step 2": assuming the evidence weighs in towards my notion, what would it take to develop a systematic approach to studying bureaucracy from the perspective of AI or even xenosapience, such that bureaucracies could be either "programmed" or communicated with directly by the human agents that comprise them (and ideally by the larger pool of human stakeholders that are forced to interact with them?)