paulfchristiano comments on After critical event W happens, they still won't believe you - Less Wrong

37 Post author: Eliezer_Yudkowsky 13 June 2013 09:59PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (104)

You are viewing a single comment's thread. Show more comments above.

Comment author: paulfchristiano 15 June 2013 12:09:02AM 4 points [-]

Regarding atomic weapons:

  • Took many years and the prospect was widely understood amongst people who knew the field (I agree that massive wartime efforts to keep things secret are something of a special case, in terms of keeping knowledge from spreading from people who know what's up to other people).
  • Once you can make nuclear weapons you still have a continuous increase in destructive power; did it start from a level much higher than conventional bombing?

I do think this example is good for your case and unusually extreme, but if we are talking about a few years I think it still isn't surprising (except perhaps because of military secrecy).

but this is not the same as anticipating a FOOM (X), endorsing the ideals of astronomical optimization (Y) and deploying the sort of policies we might consider wise for FOOM scenarios (Z).

I don't think people will suspect a FOOM in particular, but I think they are open to the possibility to the extent that the arguments suggest it is plausible. I don't think you have argued against that much.

I don't think that people will become aggregative utilitarians when they think AI is imminent, but that seems like an odd suggestion at any rate. The policies we consider wise for a FOOM scenario are those that result in people basically remaining in control of the world rather than accidentally giving it up, which seems like a goal they basically share. Again, I agree that there is likely to be a gap between what I do and what others would do---e.g., I focus more on aggregate welfare, so am inclined to be more cautious. But that's a far cry from thinking that other people's plans don't matter, or even that my plans matter much more than everyone else's taken together.