I'm not saying the situation is impossible, just really really unlikely. The AI would need to output big binary files like images, and know someone intended to decode them, and somehow get around statistical detection by AI 2 (stenography is detectable since the lowest order bits of an image are not uniformly random.)

You might have a point that it's probably not best to publish things produced by the AI on the internet. If this is a serious risk, then it could still be done safely with a small group.

I think this is a good idea, though it's not new. I have written about this at some length (jessica linked to a few examples, but much of the content here is relevant), and it's what people usually are trying to do in apprenticeship learning. I agree there is probably no realistic scenario where you would use the reduced impact machinery instead of doing this the way you describe (i.e. the way people already do it).

Having the AI try to solve the problem (rather than simply trying to mimic the human) doesn't really buy you that much, and has big costs. If the human can't solve the problem with non-negligible probability, then you simply aren't going to get a good result using this technique. And if the human can solve the problem, then you can just train on instances where the human successfully solves it. You don't save anything computationally with the conditioning.

Bootstrapping seems like the most natural way to improve performance to superhuman levels. I expect bootstrapping to work fine, if you could get the basic protocol off the ground.

The connection to adversarial networks is not really a "parallel." They are literally the same thing (modulo your extra requirement that the system do the task, which is equivalent to Jessica's quantilization proposal but which I think should definitely be replaced with bootstrapping).

I think the most important problem is that AI systems do tasks in inhuman ways, such that imitating a human entails a significant disadvantage. Put a different way, it may be harder to train an AI to imitate a human than to simply do the task. So I think the main question is how to get over that problem. I think this is the baseline to start from, but it probably won't work in general.

Overall I feel more optimistic about approval-direction than imitation for this reason. But approval-direction has its own (extremely diluted) versions of the usual safety concerns, and imitation is pretty great since it literally avoids them altogether. So if it could be fixed that would be great.

This post covers the basic idea of collecting training data with low probability online. This post describes why it might result in very low overhead for aligned AI systems.

I have a problem with calling this a "semi-open FAI problem", because even if Eliezer's proposed solution turns out to be correct, it's still a wide open problem to develop arguments that can allow us to be confident enough in it to incorporate it into an FAI design. This would be true even if nobody can see any holes in it or have any better ideas, and doubly true given that some FAI researchers consider a different approach (which assumes that there is no such thing as "reality-fluid", that everything in the multiverse just exists and as a matter of preference we do not / can not care about all parts of it in equal measure, #4 in this post) to be at least as plausible as Eliezer's current approach.

It is not totally clear why humans are this bad at math. It is almost certainly unrelated to brains computing using neurons instead of transistors.

Why do you think that? Adding numbers is highly challenging for RNNs and is a standard challenge in recent papers investigating various kinds of differentiable memory and attention mechanisms, precisely because RNNs do so badly at it (like humans).

I agree that those directions look promising.

My impression is that act-based approaches are good for human replacements, but not good for human meta-replacements. That is, if we consider the problem of fulfilling orders in an Amazon warehouse, we have a number of different problems:

1. Move a worker and a crate of goods adjacent to each other.

2. Move a single good from the crate to the order box.

3. Organize the facility to make the above two as cost-efficient as possible.

The first is already replaced by robots, the second seems like a good candidate for imitation (we need robot eyes and robot hands that are about as good as human eyes and hands, and they can work basically the same way) / low-level control theory.

But the third is a problem of cost functions, models, simuation, calculation, and possibility enumeration. It's where creativity most comes into play, and that's something I'm pessimistic about getting out of interpolative systems instead of extrapolative systems.

There are still major gains made by the reduced scope--a warehouse management system seems easier to make safe than a generic moneymaker--but I think there's a fundamental connection between the opportunity and danger of AI (the ability to extrapolate into previously unseen realms).

? I don't see why the world needs to be sufficiently convenient to allow (1). And the problem resurfaces with huge-but-bounded utilities, so invoking (2) is not enough.

Yeah I should be a bit more careful on number 4. The point is that many papers which argue that a given NN is learning "natural" representations do so by looking at what an individual hidden unit responds to (as opposed to looking at the space spanned by the hidden layer as a whole). Any such argument seems dubious to me without further support, since it relies on a sort of delicate symmetry-breaking which can only come from either the training procedure or noise in the data, rather than the model itself. But I agree that if such an argument was accompanied by justification of why the training procedure or data noise or some other factor led to the symmetry being broken in a natural way, then I would potentially be happy.

It's not really clear why you would have the searching process be more powerful than the evaluating process

Because the first supposes a powerful AI, while the second supposes an excellent evaluation process (essentially a value alignment problem solved).

Your post motivated this in part, but it's a more general issue with optimisation processes and searches.

An unbounded utility function does not literally make you "Pascal's muggable"; there are much better ways to seek infinite utility than to pay a mugger.

Have you solved that problem, then? Most people I've talked don't seem to believe it's solved.

except as an argument against taking extreme and irreversible actions on the basis of a simple model of your values that looks appealing at the moment.

The approach I presented is designed so that you can get as close to your simple model, while reducing the risks of doing so.