Do you expect me to give you the complete solution to AI right here, right now? What are you even trying to say? You seem to be arguing that FAI is impossible. How can you possibly know that? Just because you can't immediately see a solution to the problem, doesn't mean a solution doesn't exist.

I think an AI will easily be able to learn human values from observations. It will be able to build a model of humans, and predict what we will do and say. It certainly won't base all it's understanding on a stupid movie quote. The AI will know what you want.

since by that argument dogs and cats are optimization, and blue and green are optimization, and everything is optimization

I have no idea what you are talking about. Optimization isn't that vague of a word, and I tried to give examples of what I meant by it. The ability to solve problems and design technologies. Dogs and cats can't design technology. Blue and green can't design technology. Call it what you want, but to me that's what intelligence is.

And that's all that really matters about intelligence, is it's ability to do that. If you gave me a computer program that could solve arbitrary optimization problems, who cares if it can't speak language? Who cares if it isn't an agent? It would be enormously powerful and useful.

That is also why when AI is actually programmed, people will do it by trying to get something to understand language, and that will in fact result in the kind of AI that I was talking about, namely one that aims at vague goals that can change from day to day, not at paperclips.

Again this claim doesn't follow from your premise at all. AIs will be programmed to understand language... therefore they won't have goals? What?

Humans definitely have goals. We have messy goals. Nothing explicit like maximizing paperclips, but a hodge podge of goals that evolution selected for, like finding food, getting sex, getting social status, taking care of children, etc. Humans are also more reinforcement learners than pure goal maximizers, but it's the same principle.

If I train a neural network to recognize dogs, I have no way of knowing if it learned correctly. I can't look at the weights and see if they are correct dog image recognizing weights and not something else. But I can trust the process of training and validation, that the AI has learned to recognize what dogs look like.

It's a similar principle with learning human values. Of course it's more complicated than just feeding it images of dogs, but the principle of letting AIs learn models from real world data is the important part.

AIXI-tl is very inefficient. Practical AIs will not be like that, and they will not be limited to one rigid goal like that.

Your second claim doesn't follow from the first. Practical AIs will of course be different. But the basic structure of AIXI, reinforcement learning, is agnostic to the model used. It just requires some algorithm to do learning/prediction. As prediction algorithms get better and better, they will still suffer the same problems as AIXI. Unless you are proposing some totally different model of AI than reinforcement learning, that somehow doesn't suffer from these problems.

And even if you find my theory of intelligence unconvincing, one that implies that evolution is intelligent is even less convincing, since it does not respect what people actually mean by the word.

Now we are debating definitions, which is not productive.

Evolution is not typically thought of as intelligent because it's not an agent. It doesn't exist in an environment, make observations, and adjust it's model of the world, etc. I accept that evolution is not an agent. But that doesn't matter.

There is only one problem that we really care about. Optimization. This is the only thing that really matters. The advantage humans have in this world is our ability to solve problems and develop technology. The risk and benefit of superintelligent AI comes entirely from its potential to solve problems and engineer technologies better than humans.

And that's exactly what evolution does. It's an algorithm that can solve problems and design very sophisticated and efficient machines. It does the thing we care about, despite not being an agent. Whether it meets the definition of "intelligence" or not, is really irrelevant. All that matters is if it's an algorithm that can solve the types of problems we care about. There is no reason that solving problems or designing machines should require an algorithm to be an agent.

Who says we need to hardcode human values though? Any reasonable solution will involve an AI that learns what human values are. Or some other method to the control problem that makes AIs that don't want to harm or defy their creators.

But the problem is it's not just by values. It's also by wealth and intelligence and education. If you have half of the world that is really poor, and anyone that is intelligent or wealthy automatically leaves, then they will probably stay poor forever.

And also competitive tax rates have been a popular subject in politics for a long long time. "If we tax millionaires/businesses, what stops them from just leaving to another country/state/city?"

Friendly AI is an AI which maximizes human values. We know what it is, we just don't know how to build one. Yet, anyway.

I agree. I think it's very unlikely FAI could be produced from MIRI's very abstract approach. At least anytime soon.

There are some methods that may work on NN based approaches. For instance my idea for an AI that pretends to be human. In general, you can make AIs that do not have long-term goals, only short term ones. Or even AIs that don't have goals at all and just make predictions. E.g., predicting what a human would do. The point is to avoid making them agents that maximize values in the real world.

These ideas don't solve FAI on their own. But they do give a way of getting useful work out of even very powerful AIs. You could task them with coming up with FAI ideas. The AIs could write research papers, review papers, prove theorems, write and review code, etc.

I also think it's possible that RL isn't that dangerous. Reinforcement learners can't model death and don't care about self-preservation. They may try to hijack their own reward signal, but it's difficult to understand what they would do after that. E.g. if they just tweak their own RAM to have reward = +Inf, and then not do anything else. It may be harder to create a working paperclip maximizer than is commonly believed, even if we do get superintelligent AI.