lukeprog's and Louie Helm's The Singularity and Machine Ethics draft suggests some useful directions for people wondering about their career choice. Some of them are psychology-related:

On the other hand, value extrapolation approaches to machine ethics face their own challenges. Which value extrapolation algorithm should be used, and why? (Yudkowsky’s “grown up farther together” provision looks especially vulnerable.) How can one extract a coherent set of values from the complex valuation processes of the human brain, such that this set of values can be extrapolated? Whose values should be extrapolated? How much will values converge upon extrapolation (Sobel 1999; Döring and Andersen 2009)? Is the extrapolation process computationally tractable, and can it be run without doing unacceptable harm? How can extrapolated values be implemented in the goal system of a machine, and how confident can we be that the machine will retain our values during self-improvement?

These are difficult questions that demand investigation by experts in many different fields. Neuroeconomists and other cognitive neuroscientists can continue to uncover how human values are encoded and modified in the brain. Philosophers and mathematicians can develop more sophisticated value extrapolation algorithms, building on the literature concerning reflective equilibrium and “ideal preference” or “full information” theories of value. Economists, neuroscientists, and AI researchers can extend current results in choice modelling (Hess and Daly 2010) and preference elicitation (Domshlak et al. 2011) to extract preferences from human behavior and brain activity. Decision theorists can work to develop a decision theory that is capable of reasoning about decisions and values subsequent to self-modification: a “reflective” decision theory.