The Importance of Self-Doubt
[Added 02/24/14: After I got feedback on this post, I realized that it carried unnecessary negative connotations (despite conscious effort on my part to avoid them), and if I were to write it again, I would have framed things differently. See Reflections on a Personal Public Relations Failure: A Lesson in Communication for more information. SIAI (now MIRI) has evolved substantially since 2010 when I wrote this post, and the criticisms made in the post don't apply to MIRI as presently constituted.]
Follow-up to: Other Existential Risks, Existential Risk and Public Relations
Related to: Tsuyoku Naritai! (I Want To Become Stronger), Affective Death Spirals, The Proper Use of Doubt, Resist the Happy Death Spiral, The Sin of Underconfidence
In Other Existential Risks I began my critical analysis of what I understand to be SIAI's most basic claims. In particular I evaluated part of the claim
(1) At the margin, the best way for an organization with SIAI's resources to prevent global existential catastrophe is to promote research on friendly Artificial Intelligence, work against unsafe Artificial Intelligence, and encourage rational thought.
It's become clear to me that before I evaluate the claim
(2) Donating to SIAI is the most cost-effective way for charitable donors to reduce existential risk.
I should (a) articulate my reasons for believing in the importance of self-doubt and (b) give the SIAI staff an opportunity to respond to the points which I raise in the present post as well as my two posts titled Existential Risk and Public Relations and Other Existential Risks.
Yesterday SarahC described to me how she had found Eliezer's post Tsuyoku Naritai! (I Want To Become Stronger) really moving. She explained:
I thought it was good: the notion that you can and must improve yourself, and that you can get farther than you think.
I'm used to the other direction: "humility is the best virtue."
I mean, this is a big fuck-you to the book of Job, and it appeals to me.
I was happy to learn that SarahC had been positively affected by Eliezer's post. Self-actualization is a wonderful thing and it appears as though Eliezer's posting has helped her self-actualize. On the other hand, rereading the post prompted me to notice that there's something about it which I find very problematic. The last few paragraphs of the post read:
Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws. This is akin to the principle of not taking pride in confessing your ignorance; for if your ignorance is a source of pride to you, you may become loathe to relinquish your ignorance when evidence comes knocking. Likewise with our flaws - we should not gloat over how self-aware we are for confessing them; the occasion for rejoicing is when we have a little less to confess.
Otherwise, when the one comes to us with a plan for correcting the bias, we will snarl, "Do you think to set yourself above us?" We will shake our heads sadly and say, "You must not be very self-aware."
Never confess to me that you are just as flawed as I am unless you can tell me what you plan to do about it. Afterward you will still have plenty of flaws left, but that's not the point; the important thing is to do better, to keep moving ahead, to take one more step forward. Tsuyoku naritai!
There's something to what Eliezer is saying here: when people are too strongly committed to the idea that humans are fallible this can become a self-fulfilling prophecy where humans give up on trying to improve things and as a consequence remain fallible when they could have improved. As Eliezer has said in The Sin of Underconfidence, there are social pressures that push against having high levels of confidence even when confidence is epistemically justified:
To place yourself too high - to overreach your proper place - to think too much of yourself - to put yourself forward - to put down your fellows by implicit comparison - and the consequences of humiliation and being cast down, perhaps publicly - are these not loathesome and fearsome things?
To be too modest - seems lighter by comparison; it wouldn't be so humiliating to be called on it publicly, indeed, finding out that you're better than you imagined might come as a warm surprise; and to put yourself down, and others implicitly above, has a positive tinge of niceness about it, it's the sort of thing that Gandalf would do.
I have personal experience with underconfidence. I'm a careful thinker and when I express a position with confidence my position is typically well considered. For many years I generalized from one example and assumed when people express positions with confidence they've thought their positions out as well as I have. Even after being presented with massive evidence that few people think things through as carefully as I do, I persisted in granting the (statistically ill-considered) positions of others far more weight than they deserved for the very reason that Eliezer describes above. This seriously distorted my epistemology because it led to me systematically giving ill-considered positions substantial weight. I feel that I have improved on this point, but even now, from time to time I notice that I'm exhibiting irrationally low levels of confidence in my positions.
At the same time, I know that at times I've been overconfident as well. In high school I went through a period when I believed that I was a messianic figure whose existence had been preordained by a watchmaker God who planned for me to save the human race. It's appropriate to say that during this period of time I suffered from extreme delusions of grandeur. I viscerally understand how it's possible to fall into an affective death spiral.
In my view one of the central challenges of being human is to find an instrumentally rational balance between subjecting oneself to influences which push one in the direction of overconfidence and subjecting oneself to influences which push one in the direction of underconfidence.
In Tsuyoku Naritai! Eliezer describes how Orthodox Judaism attaches an unhealthy moral significance to humility. Having grown up in a Jewish household and as a consequence having had peripheral acquaintance with orthodox Judaism I agree with Eliezer's analysis of Orthodox Judaism in this regard. In the proper use of doubt, Eliezer describes how the Jesuits allegedly are told to doubt their doubts about Catholicism. I agree with Eliezer that self-doubt can be misguided and abused.
However, reversed stupidity is not intelligence. The fact that it's possible to ascribe too much moral significance to self-doubt and humility does not mean that one should not attach moral significance to self-doubt and humility. I strongly disagree with Eliezer's prescription: "Take no pride in your confession that you too are biased; do not glory in your self-awareness of your flaws."
The mechanism that determines human action is that we do what makes us feel good (at the margin) and refrain from doing what makes us feel bad (at the margin). This principle applies to all humans, from Gandhi to Hilter. Our ethical challenge is to shape what makes us feel good and what makes us feel bad in a way that incentivizes us to behave in accordance with our values. There are times when it's important to recognize that we're biased and flawed. Under such circumstances, we should feel proud that we recognize that we're biased we should glory in our self-awareness of our flaws. If we don't, then we will have no incentive to recognize that we're biased and be aware of our flaws.
We did not evolve to exhibit admirable and noble behavior. We evolved to exhibit behaviors which have historically been correlated with maximizing our reproductive success. Because our ancestral climate was very much a zero-sum situation, the traits that were historically correlated with maximizing our reproductive success had a lot to do with gaining high status within our communities. As Yvain has said, it appears that a fundamental mechanism of the human brain which was historically correlated with gaining high status is to make us feel good when we have high self-image and feel bad when we have low self-image.
When we obtain new data, we fit it into a narrative which makes us feel as good about ourselves as possible; a way conducive to having a high self-image. This mode of cognition can lead to very seriously distorted epistemology. This is what happened to me in high school when I believed that I was a messianic figure sent by a watchmaker God. Because we flatter ourselves by default, it's very important that those of us who aspire to epistemic rationality incorporate a significant element of "I'm the sort of person who engages in self-doubt because it's the right thing to do" into our self-image. If we do this, when we're presented with evidence which entails a drop in our self-esteem, we don't reject it out of hand or minimize it as we've been evolutionarily conditioned to do because wound of properly assimilating data is counterbalanced by the salve of the feeling "At least I'm a good person as evidenced by the fact that I engage in self-doubt" and failing to exhibit self-doubt would itself entail an emotional wound.
This is the only potential immunization to the disease of self-serving narratives which afflicts all utilitarians out of virtue of their being human. Until technology allows us to modify ourselves in a radical way, we cannot hope to be rational without attaching moral significance to the practice of engaging in self-doubt. As the RationalWiki's page on LessWrong says:
A common way for very smart people to be stupid is to think they can think their way out of being apes with pretensions. However, there is no hack that transcends being human...You are an ape with pretensions. Playing a "let's pretend" game otherwise doesn't mean you win all arguments, or any. Even if it's a very elaborate one, you won't transcend being an ape. Any "rationalism" that doesn't expressly take into account humans being apes with pretensions, isn't.
In Existential Risk and Public Relations I suggested that some of Eliezer's remarks convey the impression that Eliezer has an unjustifiably high opinion of himself. In the comments to the post JRMayne wrote
I think the statements that indicate that [Eliezer] is the most important person in human history - and that seems to me to be what he's saying - are so seriously mistaken, and made with such a high confidence level, as to massively reduce my estimated likelihood that SIAI is going to be productive at all.
And that's a good thing. Throwing money into a seriously suboptimal project is a bad idea. SIAI may be good at getting out the word of existential risk (and I do think existential risk is serious, under-discussed business), but the indicators are that it's not going to solve it. I won't give to SIAI if Eliezer stops saying these things, because it appears he'll still be thinking those things.
When Eliezer responded to JRMayne's comment, Eliezer did not dispute the claim that JRMayne attributed to him. I responded to Eliezer saying
If JRMayne has misunderstood you, you can effectively deal with the situation by making a public statement about what you meant to convey.
Note that you have not made a disclaimer which rules out the possibility that you claim that you're the most important person in human history. I encourage you to make such a disclaimer if JRMayne has misunderstood you.
I was disappointed, but not surprised, that Eliezer did not respond. As far as I can tell, Eliezer does have confidence in the idea that he is (at least nearly) the most important person in human history. Eliezer's silence only serves to further confirm my earlier impressions. I hope that Eliezer subsequently proves me wrong. [Edit: As Airedale points out Eliezer has in fact exhibited public self-doubt in his abilities in his posting The Level Above Mine. I find this reassuring and it significantly lowers my confidence that Eliezer claims that he's the most important person in human history. But Eliezer still hasn't made a disclaimer on this matter decisively indicating that he does not hold such a view.] The modern world is sufficiently complicated so that no human no matter how talented can have good reason to believe himself or herself to be the most important person in human history without actually doing something which very visibly and decisively alters the fate of humanity. At present, anybody who holds such a belief is suffering from extreme delusions of grandeur.
There's some sort of serious problem with the present situation. I don't know whether it's a public relations problem or if the situation is that Eliezer actually suffers from extreme delusions of grandeur, but something has gone very wrong. The majority of the people who I know who outside of Less Wrong who have heard of Eliezer and Less Wrong have the impression that Eliezer is suffering from extreme delusions of grandeur. To such people, this fact (quite reasonably) calls into question of the value of SIAI and Less Wrong. On one hand, SIAI looks like an organization which is operating under beliefs which Eliezer has constructed to place himself in as favorable a position as possible rather than with a view toward reducing existential risk. On the other hand, Less Wrong looks suspiciously like the cult of Objectivism: a group of smart people who are obsessed with the writings of a very smart person who is severely deluded and describing these writings and the associated ideology as "rational" although they are nothing of the kind.
My own views are somewhat more moderate. I think that the Less Wrong community and Eliezer are considerably more rational than the Objectivist movement and Ayn Rand (respectively). I nevertheless perceive unsettling parallels.
In the comments to Existential Risk and Public Relations, timtyler said
...many people have inflated views of their own importance. Humans are built that way. For one thing, It helps them get hired, if they claim that they can do the job. It is sometimes funny - but surely not a big deal.
I disagree with timtyler. Anything that has even a slight systematic negative impact on existential risk is a big deal.
Some of my most enjoyable childhood experiences involved playing Squaresoft RPGs. Games like Chrono Trigger, Illusion of Gaia, Earthbound, Xenogears, and the Final Fantasy series are all stories about a group of characters who bond and work together to save the world. I found these games very moving and inspiring. They prompted me to fantasize about meeting allies who I could bond with and work together with to save the world. I was lucky enough to meet one such person in high school who I've been friends with since. When I first encountered Eliezer I found him eerily familiar, as though he was a long lost brother. This is the same feeling that is present between Siegmund and Sieglinde in the Act 1 of Wagner's Die Walküre (modulo erotic connotations). I wish that I could be with Eliezer in a group of characters as in a Squaresoft RPG working to save the world. His writings such as One Life Against the World and Yehuda Yudkowsky, 1985-2004 reveal him to be a deeply humane and compassionate person.
This is why it's so painful for me to observe that Eliezer appears to be deviating so sharply from leading a genuinely utilitarian lifestyle. I feel a sense of mono no aware, wondering how things could have been under different circumstances.
One of my favorite authors is Kazuo Ishiguro, who writes about the themes of self-deception and people's attempts to contribute to society. In a very good interview Ishiguro said
I think that's partly what interests me in people, that we don't just wish to feed and sleep and reproduce then die like cows or sheep. Even if they're gangsters, they seem to want to tell themselves they're good gangsters and they're loyal gangsters, they've fulfilled their 'gangstership' well. We do seem to have this moral sense, however it's applied, whatever we think. We don't seem satisfied, unless we can tell ourselves by some criteria that we have done it well and we haven't wasted it and we've contributed well. So that is one of the things, I think, that distinguishes human beings, as far as I can see.
But so often I've been tracking that instinct we have and actually looking at how difficult it is to fulfill that agenda, because at the same time as being equipped with this kind of instinct, we're not actually equipped. Most of us are not equipped with any vast insight into the world around us. We have a tendency to go with the herd and not be able to see beyond our little patch, and so it is often our fate that we're at the mercy of larger forces that we can't understand. We just do our little thing and hope it works out. So I think a lot of the themes of obligation and so on come from that. This instinct seems to me a kind of a basic thing that's interesting about human beings. The sad thing is that sometimes human beings think they're like that, and they get self-righteous about it, but often, they're not actually contributing to anything they would approve of anyway.
[...]
There is something poignant in that realization: recognizing that an individual's life is very short, and if you mess it up once, that's probably it. But nevertheless, being able to at least take some comfort from the fact that the next generation will benefit from those mistakes. It's that kind of poignancy, that sort of balance between feeling defeated but nevertheless trying to find reason to feel some kind of qualified optimism. That's always the note I like to end on. There are some ways that, as the writer, I think there is something sadly pathetic but also quite noble about this human capacity to dredge up some hope when really it's all over. I mean, it's amazing how people find courage in the most defeated situations.
Ishiguro's quote describes how people often behave in accordance with sincere desire to contribute and end up doing things that are very different from what they thought they were doing (things which are relatively unproductive or even counterproductive). Like Ishiguro I find this phenomenon very sad. As Ishiguro hints at, this phenomenon can also result in crushing disappointment later in life. I feel a deep spiritual desire to prevent this from happening to Eliezer.
Loading…
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)
Comments (726)
FWIW, as an entrepreneur type I consider one of my top 3 key advantages the fact that I would actually appreciate it greatly if someone explained in detail why I was wasting my time with my current project. Thinking about this motivates me significantly because I haven't met any other entrepreneur types who I'd guess this is also true for.
Semi related:
I keep a big list of ideas I'd like to implement. (Start up ideas, video games ideas, math research topics.. the three things that consume me =)
Quite often I'll find out someone is working on one of these ideas, and my immediate reaction is... relief. Relief, because I found out early enough not to waste my time. But, more than that, I look at my list of ideas like an orphanage: I'm always happy when one of them finds a loving parent =p
Out of curiosity, what do you consider your other two key advantages?
I didn't actually think of 3 key advantages, just figured that would be one of the top three. Probably if I was to list others, they would be willingness to trawl through a lot of ideas before finding one and implementing it, never giving up unless it really is the rational thing to do (the flip side of the original advantage), and coding ability. (Although this guy still freaks me out: http://weblog.markbao.com/2008/how-i-built-a-webapp-in-18-hours-for-699/)
I think people often suck at following through.
A kid genius entrepreneur.. awesome. You see kid genius mathematicians, chess players, musicians, etc... but an entrepreneur, that's really different. The subject matter forces him to diversify, rather than focus in on a single skill. I'm a little inspired.
Agreed. Sometimes I see someone working on an idea I had and become even more motivated to work on it.
I'd like to vote this up as I agree with lots of the points raised, but I am not comfortable with the personal nature of this article. I'd much rather the bits personal to Eliezer be sent via email.
Probably some strange drama avoidance thing on my part. On the other hand I'm not sure Eliezer would have a problem writing a piece like this about someone else.
I've thought to myself that I have read one too many fantasy books as a kid, so the partying metaphor hits home.
I was conflicted about posting in the way that I did precisely for the reason that you describe, but after careful consideration decided that the benefits outweighed the costs, in part because Eliezer does not appear to be reading the private messages that I send him.
I have no comment to add but I will say that this is well written and researched. It also prompted a degree of self reflection on my part. At least, that's what I told myself and I feel this warm glow inside. ;)
Indeed, given how busy everyone at SIAI has been with the Summit and the academic workshop following it, it is not surprising that there has not been much response from SIAI. I was only involved as an attendee of the Summit, and even I am only now able to find time to sit down and write something in response. At any rate, as a donor and former visiting fellow, I am only loosely affiliated with SIAI, and my comments here are solely my own, although my thoughts are certainly influenced by observations of the organization and conversation with those at SIAI. I don’t have the time/knowledge to address everything in your posts, but I wanted to say a couple of things.
I don’t disagree with you that SIAI has certain public relations problems. (Frankly, I doubt anyone at SIAI would disagree with that.) There is a lot of attention and discussion at SIAI about how to best spread knowledge about existential risks and to avoid sounding like a fringe/doomsday organization in doing so. It’s true that SIAI does consider the development of a general artificial intelligence to be the most serious existential risk facing humanity. But at least from what I have seen, much of SIAI’s current approach is to seed awareness of various existential risks among audiences that are in a position to effectively join the work in decreasing that risk.
Unfortunately, gaining recognition of existential risk is a hugely difficult task. Recent books from leading intellectuals on these issues (Sir Martin Rees’s Our Final Hour and Judge Richard Posner’s Catastrophe) don’t seem to have had very much apparent impact, and their ability to influence the general public is much greater than SIAI’s. But through the Summit and various publications, awareness does seem to be gradually increasing, including among important academics like David Chalmers.
Finally, I wanted to address one particular public relations problem, or at least, public relations issue, that is evident from your criticism so far – that is, there is an (understandable) perception that many observers have that SIAI and Eliezer are essentially synonymous. In the past, this perception may have been largely accurate. I don’t think that it currently holds true, but it definitely continues to persist in many people’s minds.
Given this perception, your primary focus on Eliezer to the exclusion of the other work that SIAI does is understandable. Nor, of course, could anyone possibly deny that Eliezer is an important part of SIAI, as its founder, board member, and prominent researcher. But there other SIAI officers, board members, researchers, and volunteers, and there is other work that SIAI is trying to do. The Summit is probably the most notable example of this. SIAI-affiliated people are also working on spreading knowledge of existential risks and the need to face them in academia and more broadly. The evolution of SIAI into an organization not focused solely on EY and his research is still a work in progress; and the rebranding of the organization as such in the minds of the public has not necessarily kept pace with even that gradual progress.
As for EY having delusions of grandeur, I want to address that, although only briefly, because EY is obviously in a much better position to address any of that if he chooses to. My understanding of the video you linked to in your previous post is that EY is commenting on both 1) his ability to work on FAI research and 2) his desire to work on that research. No matter how high EY’s opinion of his ability, and it doubtless is very high, it seems to me that I have seen comments from him recognizing that there are others with equally high (or even higher) ability, e.g., The Level Above Mine. I have no doubt EY would agree that the pool of those with the requisite ability is very limited. But the even greater obstacle to someone carrying on EY’s work is the combination of that rare ability with the also rare desire to do that research and make it one’s life work. And I think that’s why EY answered the way he did. Indeed, the reference to Michael Vassar, it seems to me, primarily makes sense in terms of the desire axis, since Michael Vassar’s expertise is not in developing FAI himself, although he has other great qualities in terms of SIAI’s current mission of spreading existential risk awareness, etc.
Speaking from personal experience, the SIAI's somewhat haphazard response to people answering its outreach calls strikes me as a bigger PR problem than Eliezer's personality. The SIAI strikes me as in general not very good at effective collective action (possibly because that's an area where Eliezer's strengths are, as he admits himself, underdeveloped). One thing I'd suggest to correct that is to massively encourage collaborative posts on LW.
Agreed. I think that communication and coordination with many allies and supporters has historically been a weak point for SIAI, due to various reasons including overcommitment of some of those tasked with communications, failure to task anyone with developing or maintaining certain new and ongoing relationships, interpersonal skills being among the less developed skill sets among those at SIAI, and the general growing pains of the organization. My impression is that there has been some improvement in this area recently, but there's still room for a lot more.
More collaborative posts on LW would be great to see. There have also been various discussions about workshops or review procedures for top-level posts that seem to have generated at least some interest. Maybe those discussions should just continue in the open thread or maybe it would be appropriate to have a top-level post where people could be invited to volunteer or could find others interested in collaboration, workshops, or the like.
Thanks for pointing out "The Level Above Mine." I had not seen it before.
It seems like an implication of your post that no one is ever allowed to believe they're saving the world. Do you agree that this is an implication? If not, why not?
Not speaking for multi, but, in any x-risk item (blowing up asteroids, stabilizing nuclear powers, global warming, catastrophic viral outbreak, climate change of whatever sort, FAI, whatever) for those working on the problem, there are degrees of realism:
"I am working on a project that may have massive effect on future society. While the chance that I specifically am a key person on the project are remote, given the fine minds at (Google/CDC/CIA/whatever), I still might be, and that's worth doing." - Probably sane, even if misguided.
"I am working on a project that may have massive effect on future society. I am the greatest mind in the field. Still, many other smart people are involved. The specific risk I am worried about may or not occur, but efforts to prevent its occurrence are valuable. There is some real possibility that I will the critical person on the project." - Possibly sane, even if misguided.
"I am working on a project that will save a near-infinite number of universes. In all likelihood, only I can achieve it. All of the people - even people perceived as having better credentials, intelligence, and ability - cannot do what I am doing. All critics of me are either ignorant, stupid, or irrational. If I die, the chance of multiverse collapse is radically increased; no one can do what I do. I don't care if other people view this as crazy, because they're crazy if they don't believe me." - Clinical diagnosis.
You're doing direct, substantial harm to your cause, because you and your views appear irrational. Those who hear about SIAI as the lead dog in this effort who are smart, have money, and are connected, will mostly conclude that this effort must not be worth anything.
I believe you had some language for Roko on the wisdom of damaging the cause in order to show off how smart you are.
I'm a little uncomfortable with the heat of my comment here, but other efforts have not been read the way I intended them by you (Others appeared to understand.) I am hopeful this is clear - and let me once again clarify that I had these views before multi's post. Before. Don't blame him again; blame me.
I'd like existential risk generally to be better received. In my opinion - and I may be wrong - you're actively hurting the cause.
--JRM
What you say sounds reasonable, but I feel it's unwise for me to worry about such things. If I were to sound such a vague alarm, I wouldn't expect anyone to listen to me unless I'd made significant contributions in the field myself (I haven't).
I don't think Eliezer believes he's irreplaceable, exactly. He thinks, or I think he thinks, that any sufficiently intelligent AI which has not been built to the standard of Friendliness (as he defines it) is an existential risk. And the only practical means for preventing the development of UnFriendly AI is to develop superintelligent FAI first. The team to develop FAI needn't be SIAI, and Eliezer wouldn't necessarily be the most important contributor to the project, and SIAI might not ultimately be equal to the task. But if he's right about the risk and the solution, and his untimely demise were to doom the world, it would be because no-one else tried to do this, not because he was the only one who could.
Not that this rules out your interpretation. I'm sure he has a high opinion of his abilities as well. Any accusation of hubris should probably mention that he once told Aubrey de Grey "I bet I can solve ALL of Earth's emergency problems before you cure aging."
Multifoliaterose said this:
Note that there are qualifications on this. If you're standing by the button that ends the world, and refuse to press it when urged, or you prevent others from pressing it (e.g. Stanislav Petrov), then you may reasonably believe that you're saving the world. But no, you may not reasonably believe that you are saving the world based on long chains of reasoning based on your intuition, not on anything as certain as mathematics and logic, especially decades in advance of anything happening.
It seems like an implication of this and other assumptions made by multi, and apparently shared by you, is that no one can believe themselves to be critical to a Friendly AI project that has a significant chance of success. Do you agree that this is an implication? If not, why not?
I don't think that is really Unknowns' point. He seems to be saying that it is unreasonable to suppose that you are so important based on the facts that (a) claims that FAI will work are unsubstantiated, and (b) even if it does, multiple people are working on it, lowering the probability that any individual person will be a lynchpin.
No, I don't agree this is an implication. I would say that no one can reasonably believe all of the following at the same time with a high degree of confidence:
1) I am critical to this Friendly AI project that has a significant chance of success. 2) There is no significant chance of Friendly AI without this project. 3) Without Friendly AI, the world is doomed.
But then, as you know, I don't consider it reasonable to put a high degree in confidence in number 3. Nor do many other intelligent people (such as Robin Hanson.) So it isn't surprising that I would consider it unreasonable to be sure of all three of them.
I also agree with Tetronian's points.
I see. So it's not that any one of these statements is a forbidden premise, but that their combination leads to a forbidden conclusion. Would you agree with the previous sentence?
BTW, nobody please vote down the parent below -2, that will make it invisible. Also it doesn't particularly deserve downvoting IMO.
I would suggest that, in order for this set of beliefs to become (psychiatrically?) forbidden, we need to add a fourth item. 4) Dozens of other smart people agree with me on #3.
If someone believes that very, very few people yet recognize the importance of FAI, then the conjunction of beliefs #1 thru #3 might be reasonable. But after #4 becomes true (and known to our protagonist), then continuing to hold #1 and #2 may be indicative of a problem.
With the hint from EY on another branch, I see a problem in my argument. Our protagonist might circumvent my straitjacket by also believing 5) The key to FAI is TDT, but I have been so far unsuccessful in getting many of those dozens of smart people to listen to me on that subject.
I now withdraw from this conversation with my tail between my legs.
All this talk of "our protagonist," as well the weird references to SquareSoft games, is very off-putting for me.
Dozens isn't sufficient. I asked Marcello if he'd run into anyone who seemed to have more raw intellectual horsepower than me, and he said that John Conway gave him that impression. So there are smarter people than me upon the Earth, which doesn't surprise me at all, but it might take a wider net than "dozens of other smart people" before someone comes in with more brilliance and a better starting math education and renders me obsolete.
Candid, and fair enough.
John Conway is smarter than me, too.
Sheer curiosity, but have you or anyone ever contacted John Conway about the topic of u/FAI and asked him what the thinks about the topic, the risks associated with it and maybe the SIAI itself?
"raw intellectual power" != "relevant knowledge". Looks like he worked on some game theory, but otherwise not much relevancy. Should we ask Steven Hawking? Or take a poll of Nobel Laureates?
I am not saying that he can't be brought up to date in this kind of discussion, and has a lot to consider, but not asking him as things are indicates little.
Richard Dawkins seems to have enough power to infer the relevant knowledge from a single question.
Simply out of curiosity:
Plenty of criticism (some of it reasonable) has been lobbed at IQ tests and at things like the SAT. Is there a method known to you (or anyone reading) that actually measures "raw intellectual horsepower" in a reliable and accurate way? Aside from asking Marcello.
I was beginning to wonder if he's available for consultation.
1) can be finessed easily on its own with the idea that since we're talking about existential risk even quite small probabilities are significant.
3) could be finessed by using a very broad definition of "Friendly AI" that amounted to "taking some safety measures in AI development and deployment."
But if one uses the same senses in 2), then one gets the claim that most of the probability of non-disastrous AI development is concentrated in one's specific project, which is a different claim than "project X has a better expected value, given what I know now about capacities and motivations, than any of the alternatives (including future ones which will likely become more common as a result of AI advance and meme-spreading independent of me) individually, but less than all of them collectively."
Who else is seriously working on FAI right now? If other FAI projects begin, then obviously updating will be called for. But until such time, the claim that "there is no significant chance of Friendly AI without this project" is quite reasonable, especially if one considers the development of uFAI to be a potential time limit.
People who will be running DARPA, or Google Research, or some hedge fund's AI research group in the future (and who will know about the potential risks or be able to easily learn if they find themselves making big progress) will get the chance to take safety measures. We have substantial uncertainty about how extensive those safety measures would need to be to work, how difficult they would be to create, and the relevant timelines.
Think about resource depletion or climate change: even if the issues are neglected today relative to an ideal level, as a problem becomes more imminent, with more powerful tools and information to deal with it, you can expect to see new mitigation efforts spring up (including efforts by existing organizations such as governments and corporations).
However, acting early can sometimes have benefits that outweigh the lack of info and resources available further in the future. For example, geoengineering technology can provide insurance against very surprisingly rapid global warming, and cheap plans that pay off big in the event of surprisingly easy AI design may likewise have high expected value. Or, if AI timescales are long, there may be slowly compounding investments, like lines of research or building background knowledge in elites, which benefit from time to grow. And to the extent these things are at least somewhat promising, there is substantial value of information to be had by investigating now (similar to increasing study of the climate to avoid nasty surprises).
I wouldn't put it in terms of forbidden premises or forbidden conclusions.
But if each of these statements has a 90% of being true, and if they are assumed to be independent (which admittedly won't be exactly true), then the probability that all three are true would be only about 70%, which is not an extremely high degree of confidence; more like saying, "This is my opinion but I could easily be wrong."
Personally I don't think 1) or 3), taken in a strict way, could reasonably be said to have more than a 20% chance of being true. I do think a probability of 90% is a fairly reasonable assignment for 2), because most people are not going to bother about Friendliness. Accounting for the fact that these are not totally independent, I don't consider a probability assignment of more than 5% for the conjunction to be reasonable. However, since there are other points of view, I could accept that someone might assign the conjunction a 70% chance in accordance with the previous paragraph, without being crazy. But if you assign a probability much more than that I would have to withdraw this.
If the statements are weakened as Carl Shulman suggests, then even the conjunction could reasonably be given a much higher probability.
Also, as long as it is admitted that the probability is not high, you could still say that the possibility needs to be taken seriously because you are talking about the possible (if yet improbable) destruction of the world.
I certainly do not assign a probability as high as 70% to the conjunction of all three of those statements.
And in case it wasn't clear, the problem I was trying to point out was simply with having forbidden conclusions - not forbidden by observation per se, but forbidden by forbidden psychology - and using that to make deductions about empirical premises that ought simply to be evaluated by themselves.
I s'pose I might be crazy, but you all are putting your craziness right up front. You can't extract milk from a stone!
That's good to know. I hope multifoliaterose reads this comment, as he seemed to think that you would assign a very high probability to the conjunction (and it's true that you've sometimes given that impression by your way of talking.)
Also, I didn't think he was necessarily setting up forbidden conclusions, since he did add some qualifications allowing that in some circumstances it could be justified to hold such opinions.
To be quite clear about which of Unknowns' points I object, my main objection is to the point:
where 'I' is replaced by "Eliezer." I assign a probability of less than 10^(-9) to you succeeding in playing a critical role on the Friendly AI project that you're working on. (Maybe even much less than that - I would have to spend some time calibrating my estimate to make a judgment on precisely how low a probability I assign to the proposition.)
My impression is that you've greatly underestimated the difficulty of building a Friendly AI.
I wish the laws of argument permitted me to declare that you had blown yourself up at this point, and that I could take my toys and go home. Alas, arguments are not won on a points system.
Out of weary curiosity, what is it that you think you know about Friendly AI that I don't?
And has it occurred to you that if I have different non-crazy beliefs about Friendly AI then my final conclusions might not be so crazy either, no matter what patterns they match in your craziness recognition systems?
I agree it's kind of ironic that multi has such an overconfident probability assignment right after criticizing you for being overconfident. I was quite disappointed with his response here.
I don't understand this remark.
What probability do you assign to your succeeding in playing a critical role on the Friendly AI project that you're working on? I can engage with a specific number. I don't know if your object is that my estimate is off by a single of order of magnitude or by many orders of magnitude.
I should clarify that my comment applies equally to AGI.
I think that I know the scientific community better than you, and have confidence that if creating an AGI was as easy as you seem to think it is (how easy I don't know because you didn't give a number) then there would be people in the scientific community who would be working on AGI.
Yes, this possibility has certainly occurred to me. I just don't know what your different non-crazy beliefs might be.
Why do you think that AGI research is so uncommon within academia if it's so easy to create an AGI?
On the other hand, assuming he knows what it means to assign something a 10^-9 probability, it sounds like he's offering you a bet at 1000000000:1 odds in your favour. It's a good deal, you should take it.
Everyone is allowed to believe they're saving the world. It is two other things, both quite obvious. First, we do not say it out loud if we don't want to appear kooky. Second, if someone really believes that he is literally saving the world, then he can be sure that he has a minor personality disorder [1], regardless of whether he will eventually save the world or not. Most great scientists are eccentric, so this is not a big deal, if you manage to incorporate it into your probability estimates while doing your job. I mean, this bias obviously affects your validity estimate for each and every argument you hear against hard AI takeoff. (I don't think your debaters so far did a good job bringing up such counterarguments, but that's beside the point.)
[1] by the way, in this case (in your case) grandiosity is the correct term, not delusions of grandeur.
Stanislav Petrov had this disorder? In thinking he was making the world a safer place, Gorbachev had this disorder? It seems a stretch to me to diagnose a personality disorder based on an accurate view of the world.
Gorbachev was leading an actual superpower, so his case is not very relevant in a psychological analysis of grandiosity. At the time of the famous incident, Petrov was too busy to think about his status as a world-savior. And it is not very relevant here what he believed after saving the world.
I didn't mean to talk about an accurate view of the world. I meant to talk about a disputed belief about a future outcome. I am not interested in the few minutes while Petrov may had the accurate view that he is currently saving the world.
So you'd prohibit someone of accurate belief? I generally regard that as a reductio.
I wouldn't prohibit anything to anyone. See my reply below to cipergoth.
If a billion people buy into a 1-in-a-billion raffle, each believing that he or she will win, then every one of them has a "prohibited" belief, even though that belief is accurate in one case.
I don't think that analogy holds up.
I wasn't making an analogy. I am surprised by that interpretation. I was providing a counterexample to the claim that it is absurd to prohibit accurate beliefs. One of my raffle-players has an accurate belief, but that player's belief is nonetheless prohibited by the norms of rationality.
???
Of course. But no English speaker would utter the phrase "I will win this raffle" as a gloss for "I have a one in a billion chance to win".
I seem to have posed my scenario in a confusing way. To be more explicit: Each of my hypothetical players would assert "I will win this raffle" with the intention of accurately representing his or her beliefs about the world. That doesn't imply literal 100% certainty under standard English usage. The amount of certainty implied is vague, but there's no way it's anywhere close to the rational amount of certainty. That is why the players' beliefs are prohibited by the norms of rationality, even though one of them is making a true assertion when he or she says "I will win this raffle".
ETA: Cata deleted his/her comment. I'm leaving my reply here because its clarification of the original scenario might still be necessary.
By "(at the margin)" do you just mean that we're definitely aiming at some local optimum, and possibly missing better (but harder to reach) world/self states? If so, I agree. If not, I don't get your meaning.
Marginal thinking
A few unrelated points:
Optimising one's lifestyle for the efficient acquisition of power to enable future creation of bulk quantities of paper-clips. For example.
It would be nice if utilitarianism signified that - but in fact it is about the greatest good for the greatest number:
"Utilitarianism (also: utilism) is the idea that the moral worth of an action is determined solely by its utility in providing happiness or pleasure as summed among all sentient beings."
Not sure why this would make sense to the OP; the referenced post talks about situations where you say 'yes', Life says 'Don't this so'. I do not think EY's work hit any obvious walls that are of the showstopper kind.
This post is a pretty accurate description of me a few years ago, when I was a Singularitarian. The largest attraction of the belief system, to me, was that it implied as an AI researcher I was not just a hero, but a superhero, potentially capable of almost single-handedly saving the world. (And yes, I loved those video games too.)
What's your current position?
Appealing though the belief is, the Singularity unfortunately isn't real. Nothing is going to come along and solve our problems for us, and AI is not going to be a magical exception to the rule that developing technology is hard.
What made you switch to this position?
There was a crack (the fundamental incoherence of the Singularitian belief system) that I was having to diligently cover with wallpaper (faith in the absence of evidence) so that I wouldn't notice it. Eventually I reached a state of mind in which I could allow myself to see the crack.
My co-worker, who's much smarter than I am, sometimes solves my programming problems for me, and problems that are very hard for me are very easy for him. Is he a magical exception to your rule? If not, what about someone who's even smarter than he is?
Your coworker is human, yes? That's what I meant by "us". If God or the Galactic Mothership or a team of invisible elves or a superintelligent AI somebody created in their basement came along and solved your programming problems for you, those would be magical exceptions to my rule.
You don't seem to want to state your beliefs clearly, and I don't have the patience to write more than this one post encouraging you to do so.
Do you believe that the difficulty of developing technology depends on the mind trying to develop it only when that mind happens to be human? Or that nothing can be smarter than the smartest human? Or what?
I'm being as clear as I can without writing an essay in every comment. But I'll put it this way:
Nothing is currently smarter than the smartest human.
This is not going to change anytime soon.
While AI does have the potential to produce better tools than we have now, there are still going to be enormous gaps in the abilities of those tools. For example, suppose you had an AI that was great at writing code from formal specifications, but didn't know enough about the real world to know what code to write. Then you would have a tool that you might find useful, but that you could not sit back and let solve your problems for you. At the end of the day, the responsibility for solving your problems would still be yours. This is very different from the Singularitarian vision where creating a superintelligent AI is the last job we need to do.
Maybe you could write a full post about your views. I'd very much like to read good criticism of singularitarism, but so far your objections aren't very strong.
The core assumptions in this comment, for example, seem to be not really visible. I'm guessing the idea is something like it'd be really, really hard to do an AI that can do everything a human does, and trying to leave real-world problem-solving to subhuman AIs won't work.
But no-one's talking about going after problems in the physical world with a glorified optimizing compiler, so why do you bring up this as the main example? The starting for a lot of current AGI thinking, as far as I've understood, is to make an AI with the ability to learn and some means to interact with the world. This AI is then expected to learn to act in the world like humans learn when they grow from newborns to adults.
So is there some kind of basic difference in understanding here, when I'm thinking of AIs as learning semi-autonomous agents, and you're thinking them as, I guess, some kind of pre-programmed unchanging procedures for doing specific things?
Yes, basically my claim is that an AI of the sort you're talking about is a job for the world over timescales of generations, not for a single team over timescales of years or decades; it's hard to prove a negative, and you are right that the comments I've been making here don't -- can't -- strongly justify that claim. I'll think about whether I can put together my reasoning into a full post.
Ok, thanks. As far as I see, this is the most important core objection then.
There's actually a second big unknown too before getting into full singularitarism, whether this kind of human-equivalent AI could easily boost itself to strongly superhuman levels with any sort of ease.
But the question of just how difficult it is to build the learning baby AI is really important, and I don't have any good ideas on how to estimate it except from stuff that can be figured out from biology. The human genome gives us the number of bits that keeps passing through evolution and the general initial complexity for humans, but it's big enough that without a very good design sense trying to navigate that kind of design space would indeed take generations. Brains and learning have been evolving for a very long time, indicating that the machinery may be very elaborate to get right. Compared to this, symbolic language seems to have popped up very quickly in evolution, which gives reason to believe that once there's a robust nonverbal cognitive architecture, adding symbolic cognition capabilities isn't nearly as hard as getting the basic architecture together.
This requires two things: knowing what you want, and learning about the world.
I don't see the fundamental problem in getting an AI to learn about the world. The informal human epistemic process has been analyzed into components, and these have been formalized and implemented in ways far more powerful than an unaided human can manage. It's a lot of work to put it all together in a self-consistent package, and to give it enough self-knowledge and world-knowledge to set it in motion, and it would require a lot of computing power. But I don't see any fundamental difficulty.
What the AI wants is utterly contingent on initial conditions. But an AI that can represent the world and learn about it, can also represent just about any goal you care to give it, so there's no extra problem to solve here. (Except for Friendliness. But that is the specific problem of identifying a desirable goal, not the general problem of implementing goal-directed behavior.)
Just reviewing this basic argument reinforces the prior impression that we are already drifting towards transhuman AI and that there's no fundamental barrier in the way. We already know enough for hard work alone to get us there - I mean the hard work of tens of thousands of researchers in many fields, not one person or one group making a super-duper effort. The other factor which seals our fate is distributed computing. Even if Moore's law breaks down, computers can be networked, and there are lots of computers.
So, we are going to face something smarter than human, which means something that can outwit us, which means something that should win if its goals are ever in conflict with ours. And there is no law of nature to guarantee that its goals will be humanly benevolent. On the contrary, it seems like anything might serve as the goal of an AI, just as "any" numerical expression might be fed to a calculator for evaluation.
What we don't know is how likely it is that the first transhuman AI's goals will be bad for us. A transhuman AI may require something like the resources of a large contemporary server farm to operate, in which case it's not going to happen by accident. There is some possibility that the inherent difficulty of getting there renders it more likely that by the time you get to transhuman AI, the people working on it have thought ahead to the time when the AI is autonomous and in fact beyond stopping, and realized that it had better have "ethics". But that just means that by the time that the discipline of AI is approaching the transhuman threshold, people are probably becoming aware of what we have come to call the problem of friendliness. It doesn't mean that the problem is sure to have been solved by the time the point of no return is reached.
All in all, therefore, I conclude (1) the Singularity concept makes sense (2) it is a matter for concern in the present and near future, not the far future (3) figuring out the appropriate initial conditions for an ethical AI is the key problem to solve (4) SIAI is historically important as the first serious attempt to solve this problem.
Do you think many people here think that "something is going to come along and solve our problem for us", or that "developing AI is easy"?
Yes. In particular, the SIAI is explicitly founded on the beliefs that
Superintelligent AI will solve all our problems.
Creating same is (unlike other, much less significant technological developments) so easy that it can be done by a single team within our lifetimes.
The following summary of SIAI's position says otherwise:
http://singinst.org/riskintro/index.html
It seems you're confusing what you personally thought earlier with what SIAI currently thinks.
(Though, technically you're partly right that what SIAI folks thought when said institution was founded is closer to what you say than their current position. But it's not particularly interesting what they thought 10 years ago if they've revised their position to be much better since then.)
Ah, thanks for the update; you're right, their claims regarding difficulty and timescale have been toned down quite a bit.
That isn't really evidence that people here (currently) believe either of those. You're claiming people here believe things even though they go against some of Eliezer's writing (and I don't remember any cries of "No, Eliezer, you're wrong! Creating AI is easy!", but I might be mistaken), and even though quite a few commenters are telling you nobody here believes that.
It depends what you mean by easy and hard. From previous conversations I expect Mr Wallace is thinking something easy is doable by means of a small group over 20-30 years and hard is a couple of generations of the whole of civilizations work.
Yes, that's how I am using the terms.
I'm wondering where people said AI development was going to be easy.
And I'm wondering where it was said that superintelligent AI will solve all our problems.
Indeed. There was a post "shut up and do the impossible" for a reason!
I upvoted this, but I'm torn about this.
In your recent posts you've been slowly, carefully, thoroughly deconstructing one person. Part of me wants to break into applause at the techniques used, and learn from them, because in my whole life of manipulation I've never mounted an attack of such scale. (The paragraph saying "something has gone very wrong" was absolutely epic, to the point of evoking musical cues somewhere at the edge of my hearing. Just like the "greatly misguided" bit in your previous post. Bravo!) But another part of me feels horror and disgust because after traumatic events in my own life I'd resolved to never do this thing again.
It comes down to this: I enjoy LW for now. If Eliezer insists on creating a sealed reality around himself, what's that to me? You don't have to slay every dragon you see. Saving one person from megalomania (real or imagined) is way less important than your own research. Imagine the worst possible world: Eliezer turns into a kook. What would that change, in the grand scheme of things or in your personal life? Are there not enough kooks in AI already?
And lastly, a note about saving people. I think many of us here have had the unpleasant experience (to put it mildly) of trying to save someone from suicide. Looking back at such episodes in my own life, I'm sure that everyone involved would've been better off if I'd just hit "ignore" at the first sign of trouble. Cut and run: in serious cases it always comes to that, no exceptions. People are very stubborn, both consciously and subconsciously - they stay on their track. They will waste their life (or spend it wisely, it's a matter of perspective), but if you join the tug-of-war, you'll waste a big chunk of yours as well.
How's that for other-optimizing?
Because you were on the giving or on the receiving end of it?
Agreed; personally I de-converted myself from orthodox judaism, but I still find it crazy when people write big scholarly books debunking the bible; it's just useless a waste of energy (part of it is academic incentives).
I haven't been involved in these situations, but taking a cue from drug addicts (who incidentally have high suicide rate) most of them do not recover, but maybe 10% do. So most of the time you'll find frustration, but one in 10 you'd save a life, I am not sure if that's worthless.
The very fate of the universe, potentially. Purely hypothetically and for the sake of the discussion:
I cannot emphasise how much this is only a reply to the literal question cousin_it asked and no endorsement or denial of any of the above claims as they relate to persons real or imagined. For example it may have been good if Frodo was arrogant enough to piss off Aragorn. He may cracked it, taken the ring from Frodo and given it to Arwen. Arwen was crazy enough to give up the immortality she already had and so would be as good a candidate as any for being able to ditch a ring, without being completely useless for basically all purposes.
I suppose I could draw from that the inference that you have a rather inflated notion of the importance of what multi is doing here, ... but, in the immortal words of Richard Milhous Nixon, "That would be wrong."
More seriously, I think everyone here realizes that EY has some rough edges, as well as some intellectual strengths. For his own self-improvement, he ought to be working on those rough edges. I suspect he is. However, in the meantime, it would be best if his responsibilities were in areas where his strengths are exploited and his rough edges don't really matter. So, just what are his current responsibilities?
Convincing people that UFAI constitutes a serious existential risk while not giving the whole field of futurism and existential risk reduction a bad rep.
Setting direction for and managing FAI and UFAI-avoidance research at SIAI.
Conducting FAI and UFAI-avoidance research.
Reviewing and doing conceptual QC on the research work product.
To be honest, I don't see EY's "rough edges" as producing any problems at all with his performance on tasks #3 and #4. Only SIAI insiders know whether there has been a problem on task #2. Based on multi's arguments, I suspect he may not be doing so well on #1. So, to me, the indicated response ought to be one of the following:
A. Hire someone articulate (and if possible, even charismatic) to take over task #1 and make whatever minor adjustments are needed regarding task #2.
B. Do nothing. There is no problem!
C. Get some academic papers published so that FAI/anti-UFAI research becomes interesting to the same funding sources that currently support CS, AI, and decision theory research. Then reconstitute SIAI as just one additional research institution which is fighting for that research funding.
I would be interested in what EY thinks of these three possibilities. Perhaps for different reasons, I suspect, so would multi.
[Edited to correct my hallucination of confusing multifoliaterose with wedrifid. As a result of this edit, various comments below may seem confused. Sorry about that, but I judge that making this comment clear is the higher priority.]
Was the first (unedited) 'you' intended? If so I'll note that I was merely answering a question within a counterfactual framework suggested by the context. I haven't even evaluated what potential importance multi's post may have - but the prior probability I have for 'a given post on LW mattering significantly' is not particularly high.
I like your general analysis by the way and am always interested to know what the SIAI guys are doing along the lines of either your 1,2,3 or your A, B, C. I would seriously like to see C happen. Being able and willing to make that sort of move would be a huge step forward (and something that makes any hints of 'arrogance' seem trivial.)
I think that originally Perplexed didn't look at your comment carefully and thought that multi had written it.
I think you are right. I'm just playing the disclaimer game. Since this is a political thread there is always the risk of being condemned for supporting various positions. In this case I gave a literal answer to a rhetorical question directed at multi. Following purely social reasoning that would mean that I:
Of course that comment actually lent credence to Eliezer (hence the humor) and was rather orthogonal to multi's position with respect to arrogance.
It's not that I mind too much sticking my neck out risking a social thrashing here or there. It's just that I have sufficient capability for sticking my neck out for things that I actually do mean and for some reason prefer any potential criticism to be correctly targeted. It says something about many nerds that they value being comprehended more highly than approval.
Close. Actually, I had looked at the first part of the comment and then written my response under the delusion that wedrifid had been the OP.
I am now going to edit my comment to cleanly replace the mistaken "you" with "multi"
Veering wildly off-topic:
Come on now. Humans are immortal in Tolkien, they just sit in a different waiting room. (And technically can't come back until the End of Days™, but who cares about that.)
Alright, then, call it her permanent resident status. If real death is off the table for everyone sapient, she's still taking as big a risk as any member of the Fellowship proper.
To be sure. I was only pointing out that her "giving up immortality" was not nearly as crazy as the words "giving up immortality" might suggest in other contexts.
Er... I can't help but notice a certain humor in the idea that it's terrible if I'm self-deluded about my own importance because that means I might destroy the world.
Yes, there is is a certain humor. But I hope you did read the dot points and followed the reasoning. It, among other things, suggests a potential benefit of criticism such as multi's aside from hypothetical benefits of discrediting you should it have been the case that you were not, in fact, competent.
It's some sort of mutant version of "just because you're paranoid doesn't mean they're not out to get you".
What Eliezer said. I was arguing from the assumption that he is wrong about FAI and stuff. If he's right about the object level, then he's not deluded in considering himself important.
Which, of course, would still leave the second two dot points as answers to your question.
How so? Eliezer's thesis is "AGI is dangerous and FAI is possible". If he's wrong - if AGI poses no danger or FAI is impossible - then what do you need a Frodo for?
Edited the grandparent to disambiguate the context.
(I haven't discussed that particular thesis of Eliezer's and nor does doubting that particular belief seem to be a take home message from multi's post. The great grandparent is just a straightforward answer to the paragraph it quotes.)
But if he is wrong about FAI and stuff, then he is still deluded not specifically about considering himself important, that implication is correct, he is deluded about FAI and stuff.
Agreed.
I saved someone from suicide once. While the experience was certainly quite unpleasant at the time, if I had hit "ignore," as you suggest, she would have died. I don't think that I would be better off today if I had let her die, to say nothing of her. The fact that saving people is hard doesn't mean that you shouldn't do it!
The previous post was fine, but this one is sloppy, and I don't think it's some kind of Machiavellian plot.
Not that the two are exclusive, but this sounds an awful lot like a manic episode. I assume you gave that due consideration?
I think most of us do. Your argument for this is compelling. However, I think Eliezer was just claiming that it's possible to overdo it - at least, that's the defensible core of his insight.
I've wondered if I'm obsessed with Eliezer's writings, and whether I esteem him too highly. Answers: no, and no.
Probably true. But it's sometimes easy to be on the wrong side of an argument over small differences (of course sometimes you can be certain). I guess such "there's no harm" statements (which I've also made) are biased by a desire to be conciliatory. I don't trust people to behave well when they're annoyed at each other, so I sometimes wish they would minimize the stakes.
I doubt I know any utilitarians.
Thanks for correcting my typos.
You're welcome - I've redacted my comment so it no longer mentions them.
Well, in the category of "criticisms of SIAI and/or Eliezer", this text is certainly among the better ones. I could see this included on a "required reading list" of new SIAI employees or something.
But since we're talking about a Very Important Issue, i.e. existential risks, the text might have benefited from some closing warnings, that whatever people's perceptions of SIAI, it's Very Important that they don't neglect being very seriously interested in existential risks because of issues that they might perceive a particular organization working on the topic to have (and that it might also actually have, but that's not my focus in this comment).
I.e. if people think SIAI sucks and shouldn't be supported, they should anyway be very interested in supporting the Future of Humanity Institute at Oxford, for example. Otherwise they're demonstrating very high levels of irrationality, and with regard to SIAI, are probably just looking for plausible-sounding excuses to latch onto for why they shouldn't pitch in.
Not to say that the criticism you presented mightn't be very valid (or not; I'm not really commenting on that here), but it would be very important for people to first take care that they're contributing to the reduction of existential risks in some way, and then consider to what extent exactly a particular organization such as SIAI might be doing a sub-optimal job (since they can choose a more clear-cut case of an excellent organization for their initial contribution, i.e. Bostrom's FHI as mentioned above).
Very important to you - maybe. You aware, I presume, that for most people, the end of the world is not high on their agenda. It is evidently not "very important" to them - or they would spend more time on it.
Basic biology explains this phenomenon, as I have previously explained:
"Organisms can be expected to concentrate on producing offspring - not indulging paranoid fantasies about their whole species being wiped out!"
I feel that perhaps you haven't considered the best way to maximise your chance of developing Friendly AI if you were Eliezer Yudkowsky; your perspective is very much focussed on how you see it lookin in from the outside. Consider for a moment that you are in a situation where you think you can make a huge positive impact upon the world, and have founded an organisation to help you act upon that.
Your first, and biggest problem is getting paid. You could take time off to work on attaining a fortune through some other means but this is not a certain bet, and will waste years that you could be spending working on the problem instead. Your best bet is to find already wealthy people who can be convinced that you can change the world, that it's for the best, and that they should donate significant sums of money to you, unless you believe this is even less certain than making a fortune yourself. There's already a lot of people in the world with the requisite amount of money to spare. I think seeking donations is the more rational path.
Now, given that you need to persuade people of the importance of your brilliant new idea which no one has really been considering before, and that to most people isn't at all an obvious idea. Is the better fund seeking strategy to admit to people that you're uncertain if you'll accomplish it, and compound that on top of their own doubts? Not really. Confidence is a very strong signal that will help you persuade people that you're worth taking seriously. You asking Eliezer to be more publically doubtful probably puts him in an awkward situation. I'd be very surprised if he doesn't have some doubts, maybe he even agrees with you, but to admit to these doubts would be to lower the confidence of investors in him, which would then lower further the chance of him actually being able to accomplish his goal.
Having confidence in himself is probably also important, incidentally. Talking about doubts would tend to reinforce them, and when you're embarking upon a large and important undertaking, you want to spend as much of your mental effort and time as possible on increasing the chances that you'll bring the project about, rather than dwelling on your doubts and wasting mental energy on motivating yourself to keep working.
So how to mitigate the problem that you might be wrong without running into these problems? Well, he seems ot have done fairly well here. The SIAI has now grown beyond just him, giving further perspectives he can draw upon in his work to mitigate any shortcomings in his own analyses. He's laid down a large body of work explaining the mental processes he is basing his approaches on, which should be helpful both in recruitment for SIAI, and in letting people point out flaws or weaknesses in the work he is doing. Seems to me so far he has laid the groundwork out quite well, and now it just remains to see where he and the SIAI go from here. Importantly, the SIAI has grown to the point where even if he is not considering his doubts strongly enough, even if he becomes a kook, there are others there who may be able to do the same work. And if not there, his reasoning has been fairly well laid out, and there is no reason others can't follow their own take on what needs to be done.
That said, as an outsider obviously it's wise to consider the possibility that SIAI will never meet its goals. Luckily, it doesn't have to be an either/or question. Too few people consider existential risk at all, but those of us who do consider it can spread ourselves over the different risks that we see. To the degree which you think Eliezer and the SIAI are on the right track, you can donate a portion of your disposable income to them. To the extent that you think other types of existential risk prevention matter, you can donate a portion of that money to the Future of Humanity Institute, or other relevant existential risk fighting organisation.
A number of people have mentioned the seemingly-unimpeachable reputation of the Future of Humanity Institute without mentioning that its director, Nick Bostrom, fairly obviously has a high opinion of Eliezer (e.g., he invited him to contribute not one but two chapters to the volume on Global Catastrophic Risks). Heuristically, if I have a high opinion of Bostrom and the FHI project, that raises my opinion of Eliezer and decreases the probability of Eliezer-as-crackpot.
Unknown reminds me that Multifoliaterose said this:
This makes explicit something I thought I was going to have to tease out of multi, so my response would roughly go as follows:
Upvoted for being clever.
You've (probably) refuted the original statement as an absolute.
You're deciding not to engage the issue of hubris directly.
Does the following paraphrase your position:
Here's what I (and also part of SIAI) intend to work on
I think it's very important (and you should think so for reasons outline in my writings)
If you agree with me, you should support us
? If so, I think it's fine for you to not say the obvious (that you're being quite ambitious, and that success is not assured). It seems like some people are really dying to hear you say the obvious.
Success is not assured. I'm not sure what's meant by confessing to being "ambitious". Is it like being "optimistic"? I suppose there are people who can say "I'm being optimistic" without being aware that they are instantiating Moore's Paradox but I am not one of them.
I also disclaim that I do not believe myself to be the protagonist, because the world is not a story, and does not have a plot.
I hope that the double negative in the last sentence was an error.
I introduced the term "protagonist", because at that point we were discussing a hypothetical person who was being judged regarding his belief in a set of three propositions. Everyone recognized, of course, who that hypothetical person represented, but the actual person had not yet stipulated his belief in that set of propositions.
Interesting. I don't claim great grammatical expertise but my reading puts the last question at reasonable. Am I correct in inferring that you do not believe Eliezer's usage of "I also disclaim" to mean "I include the following disclaimer: " is valid?
Regarding 'protagonist' there is some context for the kind of point Eliezer likes to make about protagonist/story thinking in his Harry Potter fanfic. I don't believe he has expressed the concept coherently as a post yet. (I don't see where you introduced the 'protagonist' word so don't know whether Eliezer read you right. I'm just throwing some background in.)
Regarding "disclaim".
I read "disclaim" as a synonym for "deny". I didn't even consider your interpretation, but upon consideration, I think I prefer it.
My mistake (again!). :(
That's interesting. I downvoted it for being clever. It was a convoluted elaboration of a trivial technicality that only applies if you make the most convenient (for Eliezer) interpretation of multi's words. This kind of response may win someone a debating contest in high school but it certainly isn't what I would expect from someone well versed in the rationalism sequences, much less their author.
I don't pay all that much attention to what multi says (no offence intended to multi) but I pay close attention to what Eliezer does. I am overwhelmingly convinced of Eliezer's cleverness and brilliance as a rationalism theorist. Everything else, well, that's a lot more blurry.
I don't think Eliezer was trying to be clever. He replied to the only real justification multi offered for why we should believe that Eliezer is suffering from delusions of grandeur. What else is he supposed to do?
I got your reply and respect your position. I don't want to engage too much here since it would overlap with discussion surrounding Eliezer's initial reply and potentially be quite frustrating.
What I would like to see is multifoliaterose giving a considered response to the "If not, why not?" question in that link. That would give Eliezer the chance to respond to the meat of the topic at hand. Eliezer has been given a rare opportunity. He can always write posts about himself, giving justifications for whatever degree of personal awesomeness he claims. That's nothing new. But in this situation it wouldn't be perceived as Eliezer grabbing the megaphone for his own self-gratification. He is responding to a challenge, answering a request.
Why would you waste the chance to, say, explain the difference between "SIAI" and "Eliezer Yudkowsky"? Or at least give some treatment of p(someone other than Eliezer Yudkowsky is doing the most to save the world). Better yet, take that chance to emphasise the difference between p(FAI is the most important priority for humanity) and p(Eliezer is the most important human in the world).
As Graehl and wedrifid observed, Eliezer responded as if the original statement were an absolute. He applied deductive reasoning and found a reductio ad absurdum. But if, instead of an absolute, you see multifoliaterose's characterization as a reference class: "People who believe themselves to be one of the few most important in the world without having already done something visible and obvious to dramatically change it," it can lower the probability that Eliezer is, in fact, that important by a large likelihood ratio.
Whether this likelihood ratio is large enough to overcome the evidence on AI-related existential risk and the paucity of serious effort dedicated to combating it is an open question.
Even if almost everything you say here is right, it wouldn't mean that there is a high probability that if you are killed in a car accident tomorrow, no one else will think about these things (reflective decision theory and so on) in the future, even people who know nothing about you personally. As Carl Shulman points out, if it is necessary to think about these things it is likely that people will, when it becomes more urgent. So it still wouldn't mean that you are the most important person in human history.
I agree with khafra. Your response to my post is distortionary. The statement which you quote was a statement about the reference class of people who believe themselves to be the most important person in the world. The statement which you quote was not a statement about FAI.
Any adequate response to the statement which you quote requires that you engage with the last point that khafra made:
You have not satisfactorily addressed this matter.
Honestly, I don't think Eliezer would look overly eccentric if it weren't for LessWrong/Overcomingbias. Comp sci is notoriously eccentric, AI research possibly more so. The stigma against Eliezer isn't from his ideas, it isn't from his self confidence, it's from his following.
Kurzweil is a more dulled case: he has good ideas, but is clearly sensational, he has a large following, but that following isn't nearly as dedicated as the one to Eliezer (not necessarily to Eliezer himself, but to his writings and the "practicing of rationality"). And the effect? I have a visceral distaste whenever I hear someone from the Kurzweil camp say something pro-singularity. It's very easy for me to imagine that if I didn't already put stock in the notion of a singularity, that hearing a Kurzweilian talk would bias me against the idea.
Nonetheless, it may very well be the case that Kurzweil has done a net good to the singularity meme (and perhaps net harm to existential risk), spreading the idea wide and far, even while generating negative responses. Is the case with Eliezer the same? I don't know. My gut says no. Taking existential risk seriously is a much harder meme to catch than believing in a dumbed down version of the singularity.
My intuition is that Eliezer by himself, although abrasive in presentation, isn't turning people off by his self confidence and grandioseness. On the contrary, I -- and I suspect many -- love to argue with intelligent people with strong beliefs. In this sense, Eliezer's self assurance is a good bait. On the other hand, when someone with inferior debating skills goes around spurting off the message of someone else, that, to me, is purely repulsive: I have no desire to talk with those people. They're the people spouting off Aether nonsense on physics forums. There's no status to be won, even on the slim chance of victory.
Finally, aside from Eliezer as himself and Eliezer through the proxy of others, there's also Eliezer as a figurehead of SIAI. Here things are different as well, and Eliezer is again no longer merely himself. He speaks for an organisation, and, culturally, we expect serious organisations to temper their outlandish claims. Take cancer research: presumably all researchers want to cure cancer. Presumably at least some of them are optimistic and believe we actually will. But we rarely hear this, and we never hear it from organizations.
I think SIAI, and Eliezer in his capacity as a figure head, probably should temper their claims as well. The idea of existential risks from AI is already pervasive. Hollywood took care of that. What remains is a battle of credibility.
(Unfortunately, I really don't know how to go about tempering claims with the previous claims already on permanent record. But maybe this isn't as important as I think it is.)
Would you include SL4 there too? I think there were discussions there years ago (well before OB, and possibly before Kurzweil's overloaded Singularity meme complex became popular) about the perception of SIAI/Singularitarianism as a cult. (I wasn't around for any such discussions, but I've poked around in the archives from time to time. Here is one example.)
I'm inclined to think that Eliezer's clear confidence in his own very high intelligence and his apparent high estimation of his expected importance (not the dictionary-definition "expected", but rather, measured as an expected quantity the usual way) are not actually unwarranted, and only violate the social taboo against admitting to thinking highly of one's own intelligence and potential impact on the world, but I hope he does take away from this a greater sense of the importance of a "the customer is always right" attitude in managing his image as a public-ish figure. Obviously the customer is not always right, but sometimes you have to act like they are if you want to get/keep them as your customer... justified or not, there seems to be something about this whole endeavour (including but not limited to Eliezer's writings) that makes people think !!!CRAZY!!! and !!!DOOMSDAY CULT!!!, and even if is really they who are the crazy ones, they are nevertheless the people who populate this crazy world we're trying to fix, and the solution can't always just be "read the sequences until you're rational enough to see why this makes sense".
I realize it's a balance; maybe this tone is good for attracting people who are already rational enough to see why this isn't crazy and why this tone has no bearing on the validity of the underlying arguments, like Eliezer's example of lecturing on rationality in a clown suit. Maybe the people who have a problem with it or are scared off by it are not the sort of people who would be willing or able to help much anyway. Maybe if someone is overly wary of associating with a low-status yet extremely important project, they do not really intuitively grasp its importance or have a strong enough inclination toward real altruism anyway. But reputation will still probably count for a lot toward what SIAI will eventually be able to accomplish. Maybe at the point of hearing and evaluating the arguments, seeming weird or high-self-regard-taboo-violating on the surface level will only screen off people who would not have made important contributions anyway, but it does affect who will get far enough to hear the arguments in the first place. In a world full of physics and math and AI cranks promising imminent world-changing discoveries, reasonably smart people do tend to build up intuitive nonsense-detectors, build up an automatic sense of who's not even worth listening to or engaging with; if we want more IQ 150+ people to get involved in existential risk reduction, then perhaps SIAI needs to make a greater point of seeming non-weird long enough for smart outsiders to switch from "save time by evaluating surface weirdness" mode to "take seriously and evaluate arguments directly" mode.
(Meanwhile, I'm glad Eliezer says "I have a policy of keeping my thoughts on Friendly AI to the object level, and not worrying about how important or unimportant that makes me", and I hope he takes that seriously. But unfortunately, it seems that any piece of writing with the implication "This project is very important, and this guy happens, through no fault of his own, to be one of very few people in the world working on it" will always be read by some people as "This guy thinks he's one of the most important people in the world". That's probably something that can't be changed without downplaying the importance of the project, and downplaying the importance of FAI probably increases existential risk enough that the PR hit of sounding overly self-important to probable non-contributors may be well worth it in the end.)
Yes, and it's called "pattern completion", the same effect that makes people think "Singularitarians believe that only people who believe in the Singularity will be saved".
This is discussed in Imaginary Positions.
The outside view of the pitch:
Maybe there are some bits missing - but they don't appear to be critical components of the pattern.
Indeed, this time there are some extra features not invented by those who went before - e.g.:
This one isn't right, and is a big difference between religion and threats like extinction-level asteroids or AI disasters: one can free-ride if that's one's practice in collective action problems.
Also: Rapture of the Nerds, Not
I don't understand why downvote this. It does sound like an accurate representation of the outside view.
Perhaps downvoted for suggesting that the salvation-for-cash meme is a modern one. I upvoted, though.
Hmm - I didn't think of that. Maybe deathbed repentance is similar as well - in that it offers sinners a shot at eternal bliss in return for public endorsement - and maybe a slice of the will.
It may have been downvoted for the caps.
I must know, have you actually encountered people who literally think that? I'm really hoping that's a comical exaggeration, but I guess I should not overestimate human brains.
I've encountered people who think Singularitarians think that, never any actual Singularitarians who think that.
What about the recent "forbidden topic"? Surely that is a prime example of this kind of thing.
Yeah, "people who think Singularitarians think that" is what I meant.
I've actually met exactly one something-like-a-Singularitarian who did think something-like-that — it was at one of the Bay Area meetups, so you may or may not have talked to him, but anyway, he was saying that only people who invent or otherwise contribute to the development of Singularity technology would "deserve" to actually benefit from a positive Singularity. He wasn't exactly saying he believed that the nonbelievers would be left to languish when cometh the Singularity, but he seemed to be saying that they should.
Also, I think he tried to convert me to Objectivism.
Technological progress has increased weath inequality a great deal so far.
Machine intelligence probably has the potential to result in enormous weath inequality.
"It's basically a modern version of a religious belief system and there's no purpose to it, like why, why must we have another one of these things ... you get an afterlife out of it because you'll be on the inside track when the singularity happens - it's got all the trappings of a religion, it's the same thing." - Jaron here.
What about less-smart people? I mean, self-motivated idealistic genius nerds are certainly necessary for the core functions of programming an FAI, but any sufficiently large organization also needs a certain number of people who mostly just file paperwork, follow orders, answer the phone, etc. and things tend to work out more efficiently when those people are primarily motivated by the organization's actual goals rather than it's willingness to pay.
Good point. It's the people in the <130 range that SIAI needs to figure out how to attract. That's where you find people like journalists and politicians.
I find it ironic that multifoliaterose said
and then the next post, instead of delineating what he found out about other existential risks (or perhaps how we should go about doing that), is about how to save Eliezer.
"The" mechanism? Citation needed.
Better, but still unsupported and unclear. What was correlated with what?
An interesting post, well written, upvoted. Mere existence of such posts here constitutes a proof that LW is still far from Objectivism, not only because Eliezer is way more rational (and compassionate) than Ayn Rand, but mainly because the other people here are aware of dangers of cultism.
However, I am not sure whether the right way to prevent cultish behaviour (whether the risk is real or not) is to issue warning like this to the leader (or any sort of warning, perhaps). The dangers of cultism emerge from simply having a leader; whatever the level of personal rationality, being a single extraordinarily revered person in any group for any longer time probably harm's one's judgement, and the overall atmosphere of reverence is unhealthy for the group. Maybe more generally, the problem not necessarily depends on existence of a leader: if a group is too devoted to some single idea, it faces lots of dangers, the gravest thereof perhaps be separation from reality. Especially if the idea lives in an environment where relevant information is not abundant.
Therefore, I would prefer to see the community concentrate on a broader class of topics, and to continue in the tradition of disseminating rationality started on OB. Mitigating existential risk is a serious business indeed, and it has to be discussed appropriately, but we shouldn't lose perspective and become too fanatic about the issue. There were many statements written on LW in recent months or years, many of them not by EY, declaring absolute preference of existential risk mitigation above everything else; those statements I find unsettling.
Final nitpick: Gandhi is misspelled in the OP.
Thanks for correcting the misspelling!
Totally agree about LW vs. Objectivism.
The case for devoting all of your altruistic efforts to a single maximally efficient cause seems strong to me, as does the case that existential risk mitigation is that maximally efficient cause. I take it you're familiar with that case (though see eg "Astronomical Waste" if not) so I won't set it all out again here. If you think I'm mistaken, actual counter-arguments would be more useful than emotional reactions.
I don't object to devoting (almost) all efforts to a single cause generally. I do, however, object to such devotion in case of FAI and the Singularity.
If a person devotes all his efforts to a single cause, his subjective feeling of importance of the cause will probably increase and most people will subsequently overestimate how important the cause is. This danger can be faced by carefully comparing the results of one's deeds with the results of other people's efforts, using a set of selected objective criteria, or measure it using some scale ideally fixed at the beginning, to protect oneself from moving the goalposts.
The problem is, if the cause is put so far in the future and based so much on speculations, there is no fixed point to look at when countering one's own biases, and the risk of a gross overestimation of one's agenda becomes huge. So the reason why I dislike the mentioned suggestions (and I am speaking, for example, about the idea that it is a strict moral duty for everybody who can to support the FAI research as much as they can, which were implicitly present at least in the discussions about the forbidden topic) is not that I reject single-cause devotion in principle (although I like to be wary about it in most situations), but that I assign too low probability to the correctness of the underlying ideas. The whole business is based on future predictions of several tens or possibly hunderts years in advance, which is historically a very unsuccessful discipline. And I can't help but include it in that reference class.
Simultaneously, I don't accept the argument of very huge utility difference between possible outcomes, which should justify one's involvement even if the probability of success (or even probability that the effort has sense) is extremely low. Pascal-wageresque reasoning is unreliable, even if formalised, because it needs careful and precise estimation of probabilities close to 1 or 0, which humans are provably bad at.
Assuming you're right, why doesn't rejection of Pascal-like wagers also require careful and precise estimation of probabilities close to 1 or 0?
I use a heuristic which tells me to ignore Pascal-like wagers and to do whatever I would do if I haven't learned about the wager (in first approximation). I don't behave like an utilitarian in this case, so I don't need to estimate the probabilities and utilities. (I think if I did, my decision would be fairly random, since the utilities and probabilities included would be almost certainly determined mostly by the anchoring effect).
I am not sure exactly what using this heuristic entails. I certainly understand the motivation behind the heuristic:
But how do you turn that (quite rational IMO) lack of trust into an action principle? I can imagine 4 possible precepts:
Is it rationally consistent to follow all 4 precepts, or is there an inconsistency?
Which of the axioms of the Von Neumann–Morgenstern utility theorem do you reject?
I think the theorem implicitly assumes logical omniscience, and using heuristics instead of doing explicit expected utility calculations should make sense in at least some types of situations for us. The question is whether it makes sense in this one.
I think this is actually an interesting question. Is there an argument showing that we can do better than prase's heuristic of rejecting all Pascal-like wagers, given human limitations?
If I had to describe my actual choices, I don't know. No one necessarily, any of the axioms possibly. My inner decision algorithm is probably inconsistent in different ways, I don't believe for example that my choices always satisfy transitivity.
What I wanted to say is that although I know that my decisions are somewhat irrational and thus sub-optimal, in some situations, like Pascal wagers, I don't find consciously creating an utility function and to calculate the right decision to be an attractive solution. It would help me to be marginally more rational (as given by the VNM definition), but I am convinced that the resulting choices would be fairly arbitrary and probably will not reflect my actual preferences. In other words, I can't reach some of my preferences by introspection, and think that an actual attempt to reconstruct an utility function would sometimes do worse than simple, although inconsistent heuristic.
The best way to advance this goal being is probably to write an interesting top-level post.
I agree. However not everybody is able to.
This post suffers from lumping together orthogonal issues and conclusions from them. Let's consider individually the following claims:
A priori, from (8) we can conclude (9). But assuming the a priori improbable (7), (8) is a rational thing for X to conclude, and (9) doesn't automatically follow. So, at this level of analysis, in deciding whether X is overconfident, we must necessarily evaluate (7). In most cases, (7) is obviously implausible, but the post itself suggests one pattern for recognizing when it isn't:
Thus, "doing something which very visibly and decisively alters the fate of humanity" is the kind of evidence that allows to conclude (7). But unfortunately there is no royal road to epistemic rationality, we can't require this particular argument that (7) in all cases. Sometimes the argument has an incompatible form.
In our case, the shape of the argument that (7) is as follows. Assuming (2), from (3) and (4) it follows that (5), and from (1), (5) and (6) we conclude (7). Note that the only claim about a person is (4), that their work contributes to development of FAI. All the other claims are about the world, not about the person.
Given the structure of this argument for the abhorrent (8), something being wrong with the person can only affect the truth of (4), and not of the other claims. In particular, the person is overconfident if person X's work doesn't in fact contribute to FAI (assuming it's possible to contribute to FAI).
Now, the extent of overconfidence in evaluating (4) is not related to the weight of importance conveyed by the object level conclusions (1), (2) and (3). One can be underconfident about (4) and still (8) will follow. In fact, (8) is rather insensitive to the strength of assertion (4): even if you contribute to FAI a little bit, but the other object level claims hold, your work is still very important.
Finally, my impression is that Eliezer is indeed overconfident about his ability to technically contribute to FAI (4), but not to the extent this post suggests, since as I said the strength of claim (8) has nothing to do with the level of overconfidence in (4), and even small contribution to FAI is enough to conclude (8) given other object level assumptions. Indeed, Eliezer never claims that success is assured:
On the other hand, only few people are currently in the position to claim (4) to any extent. One needs to (a) understand the problem statement, (b) be talented enough, and (c) take the problem seriously enough to direct serious effort at it.
My ulterior motive to elaborating this argument is to make the situation a little bit clearer to myself, since I claim the same role, just to a smaller extent. (One reason I don't have much confidence is that each time I "level up", last time around this May, I realize how misguided my past efforts were, and how much time and effort it will take to develop the skillset necessary for the next step.) I don't expect to solve the whole problem (and I don't expect Eliezer or Marcello or Wei to solve the whole problem), but I do expect that over the years, some measure of progress can be made by mine and their efforts, and I expect other people will turn up (thanks to Eliezer's work on communicating the problem statement of FAI and new SIAI's work on spreading the word) whose contributions will be more significant.
Generally speaking, your argument isn't very persuasive unless you believe that the world is doomed without FAI and that direct FAI research is the only significant contribution you can make to saving it. (EDIT: To clarify slightly after your response, I mean to point out that you didn't directly mention these particular assumptions, and that I think many people take issue with them.)
My personal, rather uninformed belief is that FAI would be a source of enormous good, but it's not necessary for humanity to continue to grow and to overcome x-risk (so 3 is weaker); X may be contributing to the development of FAI, but not that much (so 4 is weaker); and other people engaged in productive pursuits are also contributing a non-zero amount to "save the world" (so 6 is weaker.)
As such, I have a hard time concluding that X's activity is anywhere near the "most important" using your reasoning, although it may be quite important.
The argument I gave doesn't include justification of things it assumes (that you referred to). It only serves to separate the issues with claims about a person from issues with claims about what's possible in the world. Both kinds of claims (assumptions in the argument I gave) could be argued with, but necessarily separately.
OK, I now see what your post was aimed at, a la this other post you made. I agree that criticism ought to be toward person X's beliefs about the world, not his conclusions about himself.
Your analysis is very careful and I agree with almost everything that you say.
I think that one should be hesitant to claim too much for a single person on account of the issue which Morendil raises - we are all connected. Your ability to work on FAI depends on the farmers who grow your food, the plumbers who ensure that you have access to running water, the teachers who you learned from, the people at Google who make it easier for you to access information, etc.
I believe that you (and others working on the FAI problem) can credibly hold the view that your work has higher expected value to humanity than that of a very large majority (e.g. 99.99%) of the population. Maybe higher.
I don't believe that Eliezer can credibly hold the view that he's the highest expected value human who has ever lived. Note that he has not offered a disclaimer denying the view that JRMayne has attributed to him despite the fact that I have suggested that he do so twice now.
Person X believes that their activity is more important than all other people, and that no other people can do it.
Person X also believes that only this project is likely to save the world.
Person X also believes that FAI will save the world on all axes, including political and biological.
--JRM
I'd argue that a lot of people's work does. Everybody that contributes to keeping the technological world running (from farmers to chip designers) enables us to potentially save ourselves from the longer term non-anthrogenic existential risks.
Agreed. More broadly, everyone affects anthropogenic existential risks too, which limits the number of orders of magnitude one can improve in impact from a positive start.
Obviously, you need to interpret that statement as "Any given person's work doesn't significantly contribute to saving the world". In other words, if we "subtract" that one person, the future (in the aspect of the world not ending) changes insignificantly.
Are you also amending 4) to have the significant clause?
Because there are lots of smart people that have worked on AI, whose work I doubt would be significant. And that is the nearest reference class I have for likely significance of people working on FAI.
I'm not amending, I'm clarifying. (4) doesn't have world-changing power in itself, only through the importance of FAI implied by other arguments, and that part doesn't apply to activity of most people in the world. I consider the work on AI as somewhat significant as well, although obviously less significant than work on FAI at the margain, since much more people are working on AI. The argument, as applied to their work, makes them an existential threat (moderate to high when talking about the whole profession, rather weak when talking about individual people).
As for the character of work, I believe that at the current stage, productive work on FAI is close to pure mathematics (but specifically with problem statements not given), and very much unlike most of AI or even the more rigorous kinds from machine learning (statistics).
That makes me wonder who will replace Norman Borlaug, or lets say any particular influential writer or thinker.
I actually read this as a literal, technical statement about when to let the reward modules of our minds trigger, and not a statement about whether low or high confidence is desirable. Finding a flaw in oneself is only valuable if it's followed by further investigation into details and fixes, and, as a purely practical matter, that investigation is more likely to happen if you feel good about having found a fix, than if you feel good about having found a flaw.