Thanks for this post!

>adversarial examples

Consider two strategies for developing software: top-down and bottom-up. Each strategy has its own risks. The risk of the bottom-up approach is that you build components that don't end up being useful in the final design. The risk of the top-down approach is that you make assumptions about the sort of components it's possible to build that don't end up being true. Because each approach has advantages and disadvantages, I think it makes sense for people to take both approaches when planning a large, important software project such as an FAI.

OpenAI is working on how to solve the problem of adversarial examples, which is great. But I also think it's useful for people to ask the question: suppose we had a solution to the problem of adversarial examples. Would it then be possible to build an FAI? If so, how? By answering this question, we can know things like: How useful is it for us to work on adversarial examples, relative to other bottom-up AI safety work? Are there other problems such that if we were able to solve them, and combine the solution to them with our solution to adversarial examples, we'd be able to build an FAI?

BTW, I would guess that top-down work is a comparative advantage of competitions like this one, since bottom-up work is the kind of thing that academic journals will publish.

>treacherous turns

Suppose we have an algorithm that's known to be a behavior-executor but is otherwise a black box. If it's truly an accurate imitation of human values, in the sense that the box agrees exactly with us about what kind of worlds are valuable, then maybe you'd still have the problem of a treacherous turn in the sense that once an AI using this box as its values gets powerful enough, it grabs hold of civilization's steering wheel and doesn't let go. But that wouldn't be a problem if the AI's values were determined by this hypothetical perfect black box, because it would steer us exactly where we wanted to go anyway.

>nonsentient uploads

This seems to point to a deleted post. Did you mean this post? Or did you mean to refer to your conversation with Wei Dai in the thread you linked?

>Artificial Mysterious Intelligence

I largely agree with that post, but there's a subtle distinction I want to make.

When you develop software top-down, you start with a vague idea of what you want to build, and then you break it down in to several components. For each component, you have a vague idea of how it will work--hopefully at least a little less vague than your idea about how the thing as a whole will work. The process proceeds from gathering requirements, to writing a spec, to architecture diagrams, to pseudocode, until finally you have working code. Software architecture is kinda like philosophy in a certain way.

To a software architect who's early in the design stage, "this component has not been fully specified" is not a very interesting objection. At this point, none of the components have been fully specified! More interesting objections:

• "Based on the sketch you've provided, it doesn't seem like this software would actually do what we need if it was developed"
• "I don't see how this component could be made secure"
• "I don't think we can build this component" / "This component seems at least as difficult to create as the project as a whole"

Note that the last two objections do not necessarily mean the design as a whole is dead on arrival. If the software architect knows what they're doing, they are describing a subcomponent because they think it's at least a little easier than the entire project. In this case, there's a difference of opinion which indicates that it could be useful to discuss that subcomponent more.

If FAI is rated as impossibly difficult, and we're able to reduce it to subcomponents rated at most as extremely difficult, that seems like progress. If we're able to reduce it to components that we think might become available before AGI as a whole becomes available, that creates the possibility of FAI through differential technological development.

We need to understand how the black box works inside, to make sure our version's behavior is not just similar but based on the right reasons.

I think here "black-box" can be used to refer to two different things, one to refer to things in philosophy or science which we do not fully understand yet, and also to machine learning models like neural networks that seem to capture their knowledge in ways that are uninterpretable to humans.

We will almost certainly require the use of machine learning or AI to model systems that are beyond our capabilities to understand. This may include physics, complex economic systems, the invention of new technology, or yes, even human values. There is no guarantee that a theory that describes our own values can be written down and understood fully by us.

Have you ruled out any kind of theory which would allow you to know for certain that a "black-box" model is learning what you want it to learn, without understanding everything that it has learned exactly? I might not be able to actually formally verify that my neural network has learned exactly what I want it to (i.e. by extracting the knowledge out of it and comparing it to what I already know), but maybe I have formal proofs of the algorithm it is using and so I know its knowledge will be fairly robust under certain conditions. It's basically the latter we need to be aiming for.