Why do I need to recognize Friendliness to build an FAI? I only need to know that the process used to construct it results in a friendly AI. Trying to inspect the weights of a complex neural network (or whatever) is pointless as I stated earlier. We haven't the slightest idea how alphaGo's net really works, but we can trust it to beat the best Go champions.

Evolution taught humans to eliminate competition, it taught them to be aggressive and greedy -- all human values.

Evolution also taught humans to be cooperative, empathetic, and kind.

Really your objection seems to be the whole point of CEV. A CEV wouldn't just include the values of ISIS members, but also their victims. And it would be extrapolated, to not just be their current opinions on things, but what their opinions would be if they knew more. Their values if they had more time to think about and consider issues. With those two conditions, the negative parts of human values are entirely eliminated.

Do you expect me to give you the complete solution to AI right here, right now? What are you even trying to say? You seem to be arguing that FAI is impossible. How can you possibly know that? Just because you can't immediately see a solution to the problem, doesn't mean a solution doesn't exist.

I think an AI will easily be able to learn human values from observations. It will be able to build a model of humans, and predict what we will do and say. It certainly won't base all it's understanding on a stupid movie quote. The AI will know what you want.

since by that argument dogs and cats are optimization, and blue and green are optimization, and everything is optimization

I have no idea what you are talking about. Optimization isn't that vague of a word, and I tried to give examples of what I meant by it. The ability to solve problems and design technologies. Dogs and cats can't design technology. Blue and green can't design technology. Call it what you want, but to me that's what intelligence is.

And that's all that really matters about intelligence, is it's ability to do that. If you gave me a computer program that could solve arbitrary optimization problems, who cares if it can't speak language? Who cares if it isn't an agent? It would be enormously powerful and useful.

That is also why when AI is actually programmed, people will do it by trying to get something to understand language, and that will in fact result in the kind of AI that I was talking about, namely one that aims at vague goals that can change from day to day, not at paperclips.

Again this claim doesn't follow from your premise at all. AIs will be programmed to understand language... therefore they won't have goals? What?

Humans definitely have goals. We have messy goals. Nothing explicit like maximizing paperclips, but a hodge podge of goals that evolution selected for, like finding food, getting sex, getting social status, taking care of children, etc. Humans are also more reinforcement learners than pure goal maximizers, but it's the same principle.

If I train a neural network to recognize dogs, I have no way of knowing if it learned correctly. I can't look at the weights and see if they are correct dog image recognizing weights and not something else. But I can trust the process of training and validation, that the AI has learned to recognize what dogs look like.

It's a similar principle with learning human values. Of course it's more complicated than just feeding it images of dogs, but the principle of letting AIs learn models from real world data is the important part.

AIXI-tl is very inefficient. Practical AIs will not be like that, and they will not be limited to one rigid goal like that.

Your second claim doesn't follow from the first. Practical AIs will of course be different. But the basic structure of AIXI, reinforcement learning, is agnostic to the model used. It just requires some algorithm to do learning/prediction. As prediction algorithms get better and better, they will still suffer the same problems as AIXI. Unless you are proposing some totally different model of AI than reinforcement learning, that somehow doesn't suffer from these problems.

And even if you find my theory of intelligence unconvincing, one that implies that evolution is intelligent is even less convincing, since it does not respect what people actually mean by the word.

Now we are debating definitions, which is not productive.

Evolution is not typically thought of as intelligent because it's not an agent. It doesn't exist in an environment, make observations, and adjust it's model of the world, etc. I accept that evolution is not an agent. But that doesn't matter.

There is only one problem that we really care about. Optimization. This is the only thing that really matters. The advantage humans have in this world is our ability to solve problems and develop technology. The risk and benefit of superintelligent AI comes entirely from its potential to solve problems and engineer technologies better than humans.

And that's exactly what evolution does. It's an algorithm that can solve problems and design very sophisticated and efficient machines. It does the thing we care about, despite not being an agent. Whether it meets the definition of "intelligence" or not, is really irrelevant. All that matters is if it's an algorithm that can solve the types of problems we care about. There is no reason that solving problems or designing machines should require an algorithm to be an agent.

Who says we need to hardcode human values though? Any reasonable solution will involve an AI that learns what human values are. Or some other method to the control problem that makes AIs that don't want to harm or defy their creators.

But the problem is it's not just by values. It's also by wealth and intelligence and education. If you have half of the world that is really poor, and anyone that is intelligent or wealthy automatically leaves, then they will probably stay poor forever.

And also competitive tax rates have been a popular subject in politics for a long long time. "If we tax millionaires/businesses, what stops them from just leaving to another country/state/city?"

Friendly AI is an AI which maximizes human values. We know what it is, we just don't know how to build one. Yet, anyway.