You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

XiXiDu comments on Q&A with new Executive Director of Singularity Institute - Less Wrong Discussion

26 Post author: lukeprog 07 November 2011 04:58AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (177)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 15 November 2011 10:23:37AM *  19 points [-]

I'm not sure that scientific talent is the relevant variable here. More talented folk are more likely to achieve both positive and negative outcomes.

Let's assume that all the other variables are already optimized for to minimize the risk of creating an UFAI. It seems to me that the the relationship between the ability level of the FAI team and probabilities of the possible outcomes must then look something like this:

FAI probability chart

This chart isn't meant to communicate my actual estimates of the probabilities and crossover points, but just the overall shapes of the curves. Do you disagree with them? (If you want to draw your own version, click here and then click on "Modify This Chart".)

Folk do try to estimate and reduce the risks you mentioned, and to investigate alternative non-FAI interventions.

Has anyone posted SIAI's estimates of those risks?

I would also worry that signalling issues with a diverse external audience can hinder accurate discussion of important topics

That seems reasonable, and given that I'm more interested in the "strategic" as opposed to "tactical" reasoning within SIAI, I'd be happy for it to be communicated through some other means.

Comment author: XiXiDu 15 November 2011 10:58:55AM 2 points [-]

Could you elaborate on the ability axis. Could you name some people that you perceive to be of world class ability in their field. Could you further explain if you believe that there are people who are sufficiently above that class.

For example, what about Terence Tao? What about the current SIAI team?