I don't that's a good description of the orthogonality thesis.

Probably not, but it highlights the relevant (or at least related) portion. I suppose I could have been more precise by specifying terminal values, since things like paperclips are obviously instrumental values, at least for us.

Human's don't effectively self modify

Agreed, except in the trivial case where we can condition ourselves to have different emotional responses. That's substantially less dangerous, though.

Partial optimization for human values is easier than total optimization.

Why do you believe that?

I'm not sure I do, in the sense that I wouldn't assign the proposition >50% probability. However, I might put the odds at around 25% for a Reduced Impact AI architecture providing a useful amount of shortcuts.

That seems like decent odds of significantly boosting expected utility. If such an AI would be faster to develop by even just a couple years, that could make the difference between winning and loosing an AI arms race. Sure, it'd be at the cost of a utopia, but if it boosted the odds of success enough it'd still have enough expected utility to compensate.

I'm still not sure I understand you correctly. I suspect that if we follow this to the end, we will discover that we are only arguing semantics, and don't actually disagree over anything tangible. If that's your impression too, please say so, and we'll both save ourselves some time.

I only said that it would reduce chance of stupid decisions

I wouldn't disagree that having such an operator is better than not having one. I am questioning the value of having the operator uploaded. Why would programing an AI to care about the operator's values and not manipulate the operator be easier if the operator is uploaded? Wouldn't the operator just be manipulated even faster?

The only answer I see to that is that the uploading part is just to provide a faster and better user interface. If value loading was done via a game of 20 billion questions, for example, this would take an impractically long time. (Thousands of years, if just one person at a time is answering questions.) Same goes if the AI learns values via machine learning, using rewards and punishments given out by the operator, although you'd still have to keep it from wire-heading by manipulating the operator. Also, as an interesting aside, it may be easier to pull values directly out of someone's brain.

If we're only arguing about semantics, however, I have a guess at the source:

I understand "failed FAI" to be something like a pure smile maximizer, which has just as much incentive to route around human operators as a paperclip maximizer or suffering maximizer. It wouldn't care about our values any more than we care about what sorts of values evolution tried to give us. The unstated assumption here is that value uploading failed or never happened, and the AI is no longer trying to load values, but only implement the values it has. I believe this is what you're gesturing toward with "real UFAI".

Do you understand "failed FAI" to be one which simply misunderstood our values, like a smile maximizer, but which never exited the value loading phase? This sort of AI might have some sort of "uncertainty" about it's utility function. If so, it might still care about what values we intended to give it.

I don't mean to sound dismissive, but is that any better than any other boxing technique, like requiring it to ask verbal permission of a physical operator?

Unless I'm missing something, the AI would still have all the same incentives to influence the operator's answers, and solving those problems would be just as difficult for a digital operator as a physical one.

I'm pretty sure I also mostly disagree with claim 6. (See my other reply below.)

The only specific concrete change that comes to mind is that it may be easier to take one person's CEV than aggregate everyone's CEV. However, this is likely to be trivially true, if the aggregation method is something like averaging.

If that's 1 or 2 more lines of code, then obviously it doesn't really make sense to try and put those lines in last to get FAI 10 seconds sooner, except in a sort of spherical cow in a vacuum sort of sense. However, if "solving the aggregation problem" is a couple years worth of work, maybe it does make sense to prioritize other things first in order to get FAI a little sooner. This is especially true in the event of an AI arms race.

I’m especially curious whether anyone else can come up with scenarios where a maxipok strategy might actually be useful. For instance, is there any work being done on CEV which is purely on the extrapolation procedure or procedures for determining coherence? It seems like if only half our values can easily be made coherent, and we can load them into an AI, that might generate an okay outcome.

That was pretty much my take. I get the feeling that "okay" outcomes are a vanishingly small portion of probability space. This suggests to me that the additional marginal effort to stipulate "okay" outcomes instead of perfect CEV is extremely small, if not negative. (By negative, I mean that it would actually take additional effort to program an AI to maximize for "okay" outcomes instead of CEV.)

However, I didn't want to ask a leading question, so I left it in the present form. It's perhaps academically interesting that the desirability of outcomes as a function of “similarity to CEV” is a continuous curve rather than a binary good/bad step function. However, I couldn't really see any way of taking advantage of this. I posted mainly to see if others might spot potential low hanging fruit.

I guess the interesting follow up questions are these: Is there any chance that humans are sufficiently adaptable that human values are more than just an infinitesimally small sliver of the set of all possible values? If so, is there any chance this enables an easier alternative version of the control problem? It would be nice to have a plan B.

Ah, understood. We're on the same page, then.

Ah, thanks for the clarification. I interpreted it as a reductio ad absurdum.