steven0461 comments on Thoughts on the Singularity Institute (SI) - Less Wrong

256 Post author: HoldenKarnofsky 11 May 2012 04:31AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (1270)

Comments (1270)

You are viewing a single comment's thread. Show more comments above.

Comment author: steven0461 16 May 2012 11:12:37PM *  7 points [-]

The Rule of Succession, if I'm not mistaken, assumes a uniform prior from 0 to 1 for the probability of success. That seems unreasonable; it shouldn't be extremely improbable (even before observing failure) that fewer than one in a thousand such claims result in a working AGI. So you have to adjust downward somewhat from there, but it's hard to say how much.

(This is in addition to the point that user:othercriteria makes in the sibling comment.)

Comment author: [deleted] 17 May 2012 01:18:48AM *  2 points [-]

You're correct, but where would I find a better prior? I'd rather be too conservative than resort to wild guessing (which it would be, since I'm not an expert on AGI).

(A variant of this is rhollerith_dot_com's objection below, that I failed to take into account whatever the probability of working AGI leading to death is. Presumably that changes the prior as well.)

Comment author: gwern 17 May 2012 01:38:24AM *  3 points [-]
  • Q. How can I find the priors for a problem?

  • A. Many commonly used priors are listed in the Handbook of Chemistry and Physics.

  • Q. Where do priors originally come from?

  • A. Never ask that question.

  • Q. Uh huh. Then where do scientists get their priors?

  • A. Priors for scientific problems are established by annual vote of the AAAS. In recent years the vote has become fractious and controversial, with widespread acrimony, factional polarization, and several outright assassinations. This may be a front for infighting within the Bayes Council, or it may be that the disputants have too much spare time. No one is really sure.

  • Q. I see. And where does everyone else get their priors?

  • A. They download their priors from Kazaa.

  • Q. What if the priors I want aren't available on Kazaa?

  • A. There's a small, cluttered antique shop in a back alley of San Francisco's Chinatown. Don't ask about the bronze rat.

http://yudkowsky.net/rational/bayes

Comment author: TimS 17 May 2012 01:55:16AM *  -1 points [-]

Isn't the lesson of the Quantum Physics sequence that ordinary humans today should get their priors from the least complex (and falsifiable?) statements that aren't inconsistent with empirical knowledge.

Comment author: steven0461 17 May 2012 01:37:20AM 3 points [-]

I don't know where to get a good prior. I suppose you might look at past instances where someone claimed to be close to doing something that seemed about as difficult and confusing as AGI seems to be (before taking into account a history of promises that didn't pan out, but after taking into account what we know about the confusingness of the problem, insofar as that knowledge doesn't itself come from the fact of failed promises). I don't know what that prior would look like, but it seems like it would assign (if you randomly selected a kind of feat) a substantially greater than 1/10 probability of seeing at least 10 failed predictions of achieving that feat for every successful such prediction, a substantially greater than 1/100 probability of seeing at least 100 failed predictions for every successful prediction, and so on.

Powered by Reddit Powered by Reddit