whpearson comments on The Importance of Self-Doubt - Less Wrong

23 Post author: multifoliaterose 19 August 2010 10:47PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (726)

You are viewing a single comment's thread. Show more comments above.

Comment author: Vladimir_Nesov 20 August 2010 12:05:31PM *  43 points [-]

This post suffers from lumping together orthogonal issues and conclusions from them. Let's consider individually the following claims:

  1. The world is in danger, and the feat of saving the world (if achieved) would be very important, more so than most other things we can currently do.
  2. Creating FAI is possible.
  3. Creating FAI, if possible, will be conductive to saving the world.
  4. If FAI is possible, person X's work contributes to developing FAI.
  5. Person X's work contributes to saving the world.
  6. Most people's work doesn't contribute to saving the world.
  7. Person X's activity is more important than that of most other people.
  8. Person X believes their activity is more important than that of most other people.
  9. Person X suffers from delusions of grandeur.

A priori, from (8) we can conclude (9). But assuming the a priori improbable (7), (8) is a rational thing for X to conclude, and (9) doesn't automatically follow. So, at this level of analysis, in deciding whether X is overconfident, we must necessarily evaluate (7). In most cases, (7) is obviously implausible, but the post itself suggests one pattern for recognizing when it isn't:

The modern world is sufficiently complicated so that no human no matter how talented can have good reason to believe himself or herself to be the most important person in human history without actually doing something which very visibly and decisively alters the fate of humanity.

Thus, "doing something which very visibly and decisively alters the fate of humanity" is the kind of evidence that allows to conclude (7). But unfortunately there is no royal road to epistemic rationality, we can't require this particular argument that (7) in all cases. Sometimes the argument has an incompatible form.

In our case, the shape of the argument that (7) is as follows. Assuming (2), from (3) and (4) it follows that (5), and from (1), (5) and (6) we conclude (7). Note that the only claim about a person is (4), that their work contributes to development of FAI. All the other claims are about the world, not about the person.

Given the structure of this argument for the abhorrent (8), something being wrong with the person can only affect the truth of (4), and not of the other claims. In particular, the person is overconfident if person X's work doesn't in fact contribute to FAI (assuming it's possible to contribute to FAI).

Now, the extent of overconfidence in evaluating (4) is not related to the weight of importance conveyed by the object level conclusions (1), (2) and (3). One can be underconfident about (4) and still (8) will follow. In fact, (8) is rather insensitive to the strength of assertion (4): even if you contribute to FAI a little bit, but the other object level claims hold, your work is still very important.

Finally, my impression is that Eliezer is indeed overconfident about his ability to technically contribute to FAI (4), but not to the extent this post suggests, since as I said the strength of claim (8) has nothing to do with the level of overconfidence in (4), and even small contribution to FAI is enough to conclude (8) given other object level assumptions. Indeed, Eliezer never claims that success is assured:

Success is not assured. I'm not sure what's meant by confessing to being "ambitious". Is it like being "optimistic"?

On the other hand, only few people are currently in the position to claim (4) to any extent. One needs to (a) understand the problem statement, (b) be talented enough, and (c) take the problem seriously enough to direct serious effort at it.

My ulterior motive to elaborating this argument is to make the situation a little bit clearer to myself, since I claim the same role, just to a smaller extent. (One reason I don't have much confidence is that each time I "level up", last time around this May, I realize how misguided my past efforts were, and how much time and effort it will take to develop the skillset necessary for the next step.) I don't expect to solve the whole problem (and I don't expect Eliezer or Marcello or Wei to solve the whole problem), but I do expect that over the years, some measure of progress can be made by mine and their efforts, and I expect other people will turn up (thanks to Eliezer's work on communicating the problem statement of FAI and new SIAI's work on spreading the word) whose contributions will be more significant.

Comment author: whpearson 20 August 2010 02:20:05PM 4 points [-]

Most people's work doesn't contribute to saving the world.

I'd argue that a lot of people's work does. Everybody that contributes to keeping the technological world running (from farmers to chip designers) enables us to potentially save ourselves from the longer term non-anthrogenic existential risks.

Comment author: Vladimir_Nesov 20 August 2010 02:32:29PM *  4 points [-]

Obviously, you need to interpret that statement as "Any given person's work doesn't significantly contribute to saving the world". In other words, if we "subtract" that one person, the future (in the aspect of the world not ending) changes insignificantly.

Comment author: whpearson 20 August 2010 02:46:22PM 2 points [-]

Are you also amending 4) to have the significant clause?

Because there are lots of smart people that have worked on AI, whose work I doubt would be significant. And that is the nearest reference class I have for likely significance of people working on FAI.

Comment author: Vladimir_Nesov 20 August 2010 03:04:52PM *  1 point [-]

I'm not amending, I'm clarifying. (4) doesn't have world-changing power in itself, only through the importance of FAI implied by other arguments, and that part doesn't apply to activity of most people in the world. I consider the work on AI as somewhat significant as well, although obviously less significant than work on FAI at the margain, since much more people are working on AI. The argument, as applied to their work, makes them an existential threat (moderate to high when talking about the whole profession, rather weak when talking about individual people).

As for the character of work, I believe that at the current stage, productive work on FAI is close to pure mathematics (but specifically with problem statements not given), and very much unlike most of AI or even the more rigorous kinds from machine learning (statistics).

Comment author: MartinB 20 August 2010 02:58:29PM 1 point [-]

That makes me wonder who will replace Norman Borlaug, or lets say any particular influential writer or thinker.

Comment author: CarlShulman 20 August 2010 02:23:30PM 1 point [-]

Agreed. More broadly, everyone affects anthropogenic existential risks too, which limits the number of orders of magnitude one can improve in impact from a positive start.