Very thoughtful response. Thank you for taking the time to respond even though its clear that I am painfully new to some of the concepts here.
Why on earth would anyone build any "'tangible object' maximizer"? That seems particularly foolish.
AI boxing ... fantastic. I agree. A narrow AI would not need a box. Are there any tasks an AGI can do that a narrow AI cannot?
If there is no task that a narrow AI can't do, then I'm not sure what you mean by "narrow" AI. A general AI is able to take any physically possible sequence of actions in order to accomplish its goal in unfamiliar environments. Generally that includes things a narrow AI would not be programmed to do.
One of the things an AGI can do is be set loose upon the world to accomplish some goal for perpetuity. That's what gets people here excited or scared about the prospects of AGI.
AI friendliness is an important goal and it would be insanely dangerous to build an AI without researching this issue first. I think this is pretty much the consensus view, and that is perfectly sensible.
However, I believe that we are making the wrong inferences from this.
The straightforward inference is "we should ensure that we completely understand AI friendliness before starting to build an AI". This leads to a strongly negative view of AI researchers and scares them away. But unfortunately reality isn't that simple. The goal isn't "build a friendly AI", but "make sure that whoever builds the first AI makes it friendly".
It seems to me that it is vastly more likely that the first AI will be built by a large company, or as a large government project, than by a group of university researchers, who just don't have the funding for that.
I therefore think that we should try to take a more pragmatic approach. The way to do this would be to focus more on outreach and less on research. It won't do anyone any good if we find the perfect formula for AI friendliness on the same day that someone who has never heard of AI friendliness before finishes his paperclip maximizer.
What is your opinion on this?