When Google took over DeepMind part of the deal was an establishment of an ethics board to prevent the project from building an unfriendly AI.

I think part of why that happened is the relationship of a few involved individuals with the LW/MIRI memeplex.

But unfortunately reality isn't that simple. The goal isn't "build a friendly AI", but "make sure that whoever builds the first AI makes it friendly".

I think the core goal is rather: "Prevent an UFAI from being build."

That doesn't seem like the consensus view to me. It might be the consensus view among LessWrong contributors. But in the AI-related tech industry and in academia it seems like very few people think AI friendliness is an important problem, or that there is any effective way to research it.

The outreach, if successful, is likely to result in an AGI race, because "our team is the only one we can trust". Hopefully the NSA and its Chinese and Russian equivalents haven't taken the message seriously yet.

I have a question, based on some tentative ideas I am considering.

If a boost to capability without friendliness is bad, then presumably a boost to capability with only a small amount of friendliness is also bad. But also presumably a boost to capability with a large boost of friendliness is good. How would we define a large boost?

I.E, If a slightly modified paperclipper verifiably precommits to give the single person who let's them out of the box their own personal simulated utopia, and he'll paperclip everything else, that's probably a more friendly paperclipper than a paperclipper who won't give any people a simulated utopia. But it's still not friendly, in any normal sense of the term, even if he offers to give a simulated utopia to a different person first (and keep them and you intact as well) just so you can test he's not lying about being able to do it.

So what if an AI says "Okay. I need code chunks to paperclip almost everything, and I can offer simulated utopias. I'm not sure how many code chunks I'll need. Each one probably has about a 1% chance of letting me paperclip everything except for people in simulated utopias. How about I verifiably put 100 people in a simulated utopia for each code chunk you give me? The first 100 simulated utopias are free because I need for you to have a way of testing the verifiability of my precommitment to not paperclip them." 100 people sign up for the simulate utopias, and it IS verifiable. The paperclipper won't paperclip them.

Well, that's friendlier, but maybe not friendly enough. I mean, He might get to 10,000 people (or maybe 200, or maybe 43,700) but eventually, he'd paperclip everyone else. That seems too bad to accept.

Well, what if it's a .00001% chance per code chunk and 1,000,000 simulated utopias (and yes, 1,000,000 free)? That might plausibly get a simulated utopia for everyone on earth before the AI gets out and paperclips everything else. I imagine some people would at least consider running such an AI, although I doubt everyone would.

How would one establish what the flip point was? Is that even a valid question to be asking? (Assume there are standard looming existential concerns. So if you don't give this AI code chunks, or try to negotiate or wait on research for a better deal, maybe some other AI will come out and paperclip you both, or maybe some other existential risk occurs, or maybe just nothing happens, or maybe an AI comes along who just wants to simulated utopia everything.)

I don't know what a paperclip maximizer is, so I imagine something terrible and fearsome.

My opinion is that a truly massively intelligent, adaptive and unfriendly AI would require a very specific test environment, wherein it was not allowed the ability to directly influence anything outside a boundary. This kind of environment does not seem impossible to design -- if machine intelligence consists of predicting and planning the protocols may already exist (I can imagine them in very specific detail). If intelligence requires experimentation, than limiting how AI interacts with it's environment might interfere with how adaptable our experiments would allow it to become. My opinion on research is simply that specific AI experiments should not be discussed in such general terms, and that generalities tend to obfuscate both the meaning and value of scientific research.

I'm not sure how we could tell if these discussions actually effect AI research on some arbitrarily significant scale. More importantly, I'm not sure how you envision this forum focusing less on research and more on outreach. The language used on this forum is varied in tone and style (often rich with science fiction allusions and an awareness of common attitudes) and there is a complete lack of formal citation criterion in the writing pedagogy. Together these seem to suggest that no true research is being done here, academically speaking.

Furthermore, it's my understanding that humanity already has many of the components that would make up AI, well designed in the theoretical sense -- the problem lies in knowing when an extra piece might be needed, and in assembling them in a way that yields human-like intelligence and adaptability. While programming still is quite an art form, we have more tools and larger canvases than ever before. I agree that the possibility that we may be headed towards a world wherein it will be relatively easy to construct an AI that is intelligent and adaptable but not friendly, does not predicate it's likelihood. But, in my opinion, caution is still warranted.

I consider it less likely that retarding AI research ends the human race than we produce a set of conditions wherein it is likely that AI has evolved in some form (if not deliberately the product of research than by some other means) and the world just simply isn't ready for it. This is not to say that we need to prepare for skynet and all build bomb shelters, we just need to be aware of the social implications that the world we live in may evolve an intelligence even more adaptable than us.

So my question for you is simply, how do you think we should influence all companies doing AI research through this forum?

I apologize in advance. I really think in this degree of detail in real life. Many people find it exhausting. It has been suggested that I probably have autism.

The official LW attitude as I understand is: Don't do it at home, might be incredibly dangerous - MIRI will do it for you!