Some thoughts on AI, Philosophy, and Safety

paulfchristiano

I've spent some time over the last two weeks thinking about problems around FAI. I've committed some of these thoughts to writing and put them up here.

There are about a dozen real posts and some scraps. I think some of this material will be interesting to certain LWers; there is a lot of discussion of how to write down concepts and instructions formally (which doesn't seem so valuable in itself, but it seems like someone should do it at some point) some review and observations on decision theory, and some random remarks on complexity theory, entropy, and prediction markets.

I've spent some time over the last two weeks thinking about problems around FAI. I've committed some of these thoughts to writing and put them up here.

I sure hope you have a tendency to eventually converge to something that makes sense to me... Do you agree that what you post there is the product of an "initial exploration" phase that would get significantly revised and mostly discarded on the scale of months? (I had a blog just 1.5 years ago that I currently see this way, but didn't at the time...)

Have you seen Paul's latest post yet? It seems much more well formed than his previous posts on the subject.

I left a comment there, but it's still under moderation, so I'll copy it here.

For example, if we suppose that the U-maximizer can carry out any reasoning that we can carry out, then the U-maximizer knows to avoid anything which we suspect would be bad according to U (for example, torturing humans).

This seems like a problematic part of the argument. The reason we think torturing humans would be bad according to U is that we have an informal model ... (read more)

5paulfchristiano15y

Could you clarify your remarks? This seems to be a source of persistent mild disagreement, but I'm not really sure what it is. I am aware of some inferential distance between us on what seem to me to be technical aspects of decision theory, but your comments either require some misunderstanding or some other not-yet-identified inferential chasm. In the future I expect to feel basically the same way about this writing, particularly the stuff in the category "Formal Definitions," as I do today about these posts: not safe for use, but important for someone to think about and describe, if only to see more exactly why they are dangerous approaches. I expect the formal assertions I've made, to the extent I've made formal assertions, to continue to look reasonable. I am open to the possibility that there may be surprises. (For example, when making those old LW posts I didn't yet understand exactly how weird TDT agents' behavior might look to someone used to thinking in terms of CDT, so while the cryptographic boxing stuff still holds up fine the manipulation of boxed AIs doesn't; the new work can be expressed with much less wiggle room, but there may still be surprises. )

17

Some thoughts on AI, Philosophy, and Safety

17

17

17

Some thoughts on AI, Philosophy, and Safety

17

17