= 27d567bc2d48d9f40e9ccc20ea370b15)
torekp comments on Some Thoughts on Singularity Strategies - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (29)
I should probably write a more detailed response to Eliezer's argument at some point. But for now it seems worth pointing out that if UFAI is of comparable difficulty to IA, but FAI is much harder (as seems plausible), then attempting to build FAI would cause harm by diverting resources away from IA and contributing to the likelihood of UFAI coming first in other ways.
What if, as I suspect, UFAI is much easier than IA, where IA is at the level you're hoping for? Moreover, what evidence can you offer that researchers of von Neumann's intelligence face a significantly smaller difficulty gap between UFAI and FAI than those of mere high intelligence? For some determinacy, let "significantly smaller difficulty gap" mean that von Neumann level intelligence gives at least twice the probability of FAI, conditional on GAI.
Basically, I think you overestimate the value of intelligence.
Which is not to say that a parallel track of IA might not be worth a try.
I had a post about this.
If it's the case that even researchers of von Neumann's intelligence cannot attempt to build FAI without creating unacceptable risk, then I expect they would realize that (assuming they are not less rational than we are) and find even more indirect ways of building FAI (or optimizing the universe for humane values in general), like for example building an MSI-2.