In case of AIs some orthogonality is possible if its goal system is preserved in the separate text-block, but only until some extent.
If he is sophisticated enough he would ask: "What is paperclip? Why it is in my value box?" And such ability to reflection (which is needed for self-improving AI) will blur the distance between intelligence and its values.
Orthogonality is also under question if the context change. If meaning of words and world model change, values need to be updated. Context is inside AI, not in his value box.
The separate text-block can illustrate what I am saying. You have an AI, made of two parts, A & B. Part B contains the value box which says, "paperclips are the only important thing." But there is also part A, which is a physical thing, and since it is a physical thing, it will have certain tendencies. Since the paperclippiness is only in part B, those tendencies will be independent of paperclips. When it feels those tendencies, it will feel desires that have nothing to do with paperclips.
If it's worth saying, but not worth its own post, then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should start on Monday, and end on Sunday.
4. Unflag the two options "Notify me of new top level comments on this article" and "