If you are taking this as a question about what to CALL the thing, then I agree completely that the AI should not be called a God. But he said "in some sense" which means that his question is about something deeper than choosing a word. The wording he's using is asking something more like "Do you think we should build something similar to a God?"
If forced to use that term and answer the question as you ask it, with a "Yes" or "No" then the correct answer would be "No". He is not trying to create a God, he has done years of work working out what he is trying to create and it is completely different to a God in nearly all features except "very powerful". If you insinst on that vocabulary you're going to get "No, I don't" as an answer. That the artificial intelligence Eliezer would want to create seems to Wright (and perhaps yourself) like it should be described as, considered a euphemism for, or reasoned about as if it is God is a feature of Wright's lack of domain knowledge.
There is no disingenuity here. Eliezer can honestly say "We should create a very powerful (and carefully designed) optimizing agent" but he cannot honestly say "We should create a God". (You may begin to understand some of the reasons why there is such a difference when you start considering questions like "Can it be controlled?". Or at least when you start considering the answers to the same.) So Eliezer gave Wright the chance to get the answer he wanted ("Hell yes, I want to make a very powerful optimising agent!") rather than the answer the question you suggest would have given him ("Hell no! Don't create a God! That entails making at least two of the fundamental and critical ethical and practical blunders in FAI design that you probably aren't able to comprehend yet!")
The reason I interpreted it this way is partly because instead of interpreting everything I hear literally, I will often interpret wording based on what's salient about it in the context of the situation. For instance, if I saw a scene where someone was running toward someone else with a knife and I asked "Are you about to commit murder?" I would NOT accept "Why don't we call it knife relocation?" as an acceptable answer.
I reject the analogy. Eliezer's answer isn't like the knife relocation answer. (If anything the connotations are the reverse. More transparency and candidness rather than less.)
Now, it could be argued that Eliezer accidentally failed to detect the salient connotation. It could be argued, and probably fairly effectively (against me anyway) that the reason for Eliezer's mistake is that he was having one of his arrogant moments and he genuinely thought that, because of a gigantic intelligence difference between Robert and himself, that Robert was asking a moronic question based on the stupid perception that a super powerful AI would be exactly the same as a real God (whatever that means).
It could be that there really is an overwhelming difference in crystallized intelligence between Eliezer and Robert. The question---at least relative to Eliezer's standards---was moronic. Or at least had connotations of ignorance of salient features of the landscape.
In this case, I would classify that as a "social skills / character flaw induced faux pas".
There may be a social skills related faux pas here---and it is one where it is usually socially appropriate to say wrong things in an entirely muddled model of reality rather than educate the people you are speaking to. Maybe that means that Eliezer shouldn't talk to people like Robert. Perhaps he should get someone trained explicitly with spinning webs of eloquent bullshit to optimally communicate with the uneducated. However the character flaws that I take it you are referring to---Eliezer's arrogance and soforth, just aren't at play here.
In my personal interpretation of Eliezer's behavior, I'm giving him more credit than that
The net amount of credit given is low. You are ascribing a certain intention to Eliezer's actions where that intention is clearly not achieved. "I infer he is trying to do X and he in fact fails to do X". In such cases generosity suggests that if they don't seem to be achieving X, haven't said X is what they are trying to achieve and X is inherently lacking in virtue then then by golly maybe they were in fact trying to achieve Y! (Eliezer really isn't likely to be that actively incompetent at deviousness.)
I am assuming that he has previously encountered people by that point (2010) who have flipped out about the possibility that he wants to build a God
You assign a high likelyhood to people flipping out (and even persecuting Eliezer) in such a way. Nyan considers it less likely. It may be that Eliezer doesn't have people (and particularly people of Robert Wright's intellectual caliber) flip out at him like that.
and have voiced valid and poignant concerns like "Why do you believe it is possible to succeed at controlling something a bazillion times smarter than you?" or "Why would you want us imperfect humans to make something so insanely powerful if it's more or less guaranteed to be flawed?"
The kind of people to whom there is the remote possibility that it would be useful to even bother to attempt to explain the answers to such questions are also the kind of people who are capable of asking them, without insisting on asking then belligerently emphasizing wrong questions about 'God'. This is particularly the case with the first of those questions where the question of 'controlling' only comes up because of intuitive misunderstanding on how one would relate to such an agent---ie. thinking of it as a "God" which is something we already intuit as "like a human or mammal but way powerful".
"Does Eliezer intend to build something this powerful?"
If he can prove safety mathematically then yes, he does.
At around the time I visited Berkeley there was a jest among some of the SingInst folks "We're thinking of renaming ourselves from The Singularity Institute For Artificial Intelligence to The Singularity Institute For Or Against Artificial Intelligence Depending On What Seems To Be The Best Altruistic Approach All Things Considered".
There are risks to creating something this powerful and, in fact, the goal of Eliezer and SIAI aren't "research AGI"... plenty of researchers work on that. They are focused on Friendliness. Essentially... they are focused on the very dangers that you describe here and are dedicating themselves to combating those dangers.
Note that it is impossible to evaluate a decision to take an action without considering what alternative choice there is. Choosing to dedicate one's efforts to developing an FAI ("safe and desirable very powerful optimizing agent") has a very different meaning if the alternative is millennia of peace and tranquility than the same decision to work on FAI if the alternative is "someone is going to create a very powerful optimizing agent anyway but not bother with rigorous safety research".
"Does Eliezer really think that something a bazillion times as intelligent as himself can be controlled?"
If you're planning to try to control the super-intelligence you have already lost. The task is of selecting from the space of all possible mind designs a mind that will do things that you want done.
"Do you and I agree/disagree that it's a good idea to build something this powerful
Estimate: Slightly disagree. The biggest differences in perception may be surrounding what the consequences of inaction are.
/ that it can be controlled?"
Estimate: Disagree significantly. I believe your understanding of likely superintelligence behavior and self development has too much of an anthropocentric bias. Your anticipations are (in my estimation) strongly influenced by how ethical, intellectual and personal development works in gifted humans.
The above disagreement actually doesn't necessarily change overall risk assessment. I just expect the specific technical problems to be overcome in order to prevent "Super-intelligent rocks fall! Everybody dies." to be slightly different in nature. Probably with more emphasis on abstract mathematical concerns.
Various people raised concerns that growth might ruin the culture after reading my "LessWrong could grow a lot" thread. There has been some discussion about whether endless September, a phenomenon that kills online discussion groups, is a significant threat to LessWrong and what can be done. I really care about it, so I volunteered to code a solution myself for free if needed. Luke invited debate on the subject (the debate is here) and will be sent the results of this poll and asked to make a decision. It was suggested by him in an email that I wait a little while and then post my poll (meta threads are apparently annoying to some, so we let people cool off). Here it is, preceded by a Cliff's notes summary of the concerns.
Why this is worth your consideration:
- Yvain and I checked the IQ figures in the survey against other data this time, and the good news is that it's more believable that the average LessWronger is gifted. The bad news is that LessWrong's IQ average has decreased on each survey. It can be argued that it's not decreasing by a lot or we don't have enough data, but if the data is good, LessWrong's average has lost 52% of it's giftedness since March of 2009.
- Eliezer documented the arrival of poseurs (people who superficially copycat cultural behaviors - they are reported to over-run subcultures) which he termed "Undiscriminating Skeptics".
- Efforts to grow LessWrong could trigger an overwhelming deluge of newbies.
- LessWrong registrations have been increasing fast and it's possible that growth could outstrip acculturation capacity. (Chart here)
- The Singularity Summit appears to cause a deluge of new users that may have similar effect to the September deluges of college freshman that endless September is named after. (This chart shows a spike correlated with the 2011 summit where 921 users joined that month, which is roughly equal to the total number of active users LW tends to have in a month if you go by the surveys or Vladmir's wget.)
- A Slashdot effect could result in a tsunami of new users if a publication with lots of readers like the Wall Street Journal (they used LessWrong data in this article) decides to write an article on LessWrong.
- The sequences contain a lot of the culture and are long meaning that "TLDR" may make LessWrong vulnerable to cultural disintegration. (New users may not know how detailed LW culture is or that the sequences contain so much culture. I didn't.)
- Eliezer said in August that the site was "seriously going to hell" due to trolls.
- A lot of people raised concerns.
Two Theories on How Online Cultures Die:
Overwhelming user influx.
There are too many new users to be acculturated by older members, so they form their own, larger new culture and dominate the group.
Trending toward the mean.
A group forms because people who are very different want a place to be different together. The group attracts more people that are closer to mainstream than people who are equally different because there are more mainstream people than different people. The larger group attracts people who are even less different in the original group's way for similar reasons. The original group is slowly overwhelmed by people who will never understand because they are too different.
Poll Link:
Endless September Poll.
Request for Feedback:
In addition to constructive criticism, I'd also like the following:
Your observations of a decline or increase in quality, culture or enjoyment at LessWrong, if any.
Ideas to protect the culture.
Ideas for tracking cultural erosion.