Rain comments on The $125,000 Summer Singularity Challenge - Less Wrong

20 Post author: Kaj_Sotala 29 July 2011 09:02PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (259)

You are viewing a single comment's thread. Show more comments above.

Comment author: jsteinhardt 30 July 2011 10:47:16AM 5 points [-]

Well for instance, certain approaches to AGI are more likely to lead to something friendly than other approaches are. If you believe that approach A is 1% less likely to lead to a bad outcome than approach B, then funding research in approach A is already compelling.

In my mind, a well-reasoned statistical approach with good software engineering methodologies is the mainstream approach that is least likely to lead to a bad outcome. It has the advantage that there is already a large amount of related research being done, hence there is actually a reasonable chance that such an AGI would be the first to be implemented. My personal estimate is that such an approach carries about 10% less risk than an alternative approach where the statistics and software are both hacked together.

In contrast, I estimate that SIAI's FAI approach would carry about 90% less risk if implemented than a hacked-together AGI. However, I assign very low probability to SIAI's current approach succeeding in time. I therefore consider the above-mentioned approach more effective.

Another alternative to SIAI that doesn't require estimates about any specific research program would be to fund the creation of high-status AI researchers who care about Friendliness. Then they are free to steer the field as a whole towards whatever direction is determined to carry the least risk, after we have the chance to do further research to determine that direction.

Comment author: Rain 30 July 2011 03:47:39PM 3 points [-]

I noticed you didn't name anybody. Did you have specific programs or people in mind?

We already seem to (roughly) agree on probabilities.

Comment author: jsteinhardt 02 August 2011 09:14:21PM *  2 points [-]

The only specific plan I have right now is to put myself in a position to hire smart people to work on this problem. I think the most robust way to do this is to get a faculty position somewhere, but I need to consider the higher relative efficiency of corporations over universities some more to figure out if it's worthwhile to go with the higher-volatility route of industry.

Also, as Paul notes, I need to consider other approaches to x-risk reduction as well to see if I can do better than my current plan. The main argument in favor of my current plan is that there is a clear path to the goal, with only modest technical hurdles and no major social hurdles. I don't particularly like plans that start to get fuzzier than that, but I am willing to be convinced that this is irrational.

EDIT: To be more explicit, my current goal is to become one of said high-status AI researchers. I am worried that this is slightly self-serving, although I think I have good reason to believe that I have a comparative advantage at this task.