Are there ‘categorical’ oughts that do not depend on an “If you want X” clause? Naturalists tend to deny this possibility, but perhaps categorical epistemic or moral oughts can be derived from the mathematics of game theory and decision theory, as naturalist Gary Drescher suggests in Good and Real. If so, it may be wise to make sure they are included in friendliness content theory, so that an AI can respect them.
The phrasing "it may be wise to make sure they are included in friendliness content theory" makes it sound wrong to my ear, as if you are brewing some kind of informal eclectic theoretical soup.
It struck me as sort of a sly understatement, which went over well compared to the more familiar "I am politely and quietly screaming that this is important" tone.
Yesterday I sat down with Lukeprog for a few hours and we produced this ten-page interview for the Singularity Institute blog. This interview contains information about the Singularity Institute's technical research program and recent staff changes that hasn't been announced anywhere else! We hope you find it informative.
http://singinst.org/blog/2011/09/15/interview-with-new-singularity-institute-research-fellow-luke-muehlhuaser-september-2011/