This general problem has been studied by Stuart Russell, Andrew Ng, and others. It's called "Inverse Reinforcement Learning", and the general idea is to learn the utility function of an agent A given training data which includes A's actual decisions, and then use that to infer an approximation of A's utility function for use in a RL agent B, where B can satisfy A's goals, perhaps better than A itself (by thinking faster and or predicting the future better).

You need to start with some sensible prior's over A's utility function for the problem to be well formed, but after that it becomes a machine learning problem.