= 27d567bc2d48d9f40e9ccc20ea370b15)
Vladimir_Nesov comments on AI Risk and Opportunity: A Strategic Analysis - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (161)
Why do you say this is a disagreement? Who at SIAI thinks FAI is likely to be implementable on time (and why)?
Right, assuming we can find any alternative avenues of comparable probability of success. I think it's unlikely for FAI to be implementable both "on time" (i.e. by humans in current society), and via alternative avenues (of which fast WBE humans seems the most plausible one, which argues for late WBE that's not hardware-limited, not pushing it now). This makes current research as valuable as alternative routes despite improbability of current research's success.
Let me rephrase: I think the expected gain from pursuing FAI is less that pursuing other methods. Other methods are less likely to work, but more likely to be implementable. I think SIAI disagrees with this accessment.
I assume that by "implementable" you mean that it's an actionable project, that might fail to "work", i.e. deliver the intended result. I don't see how "implementability" is a relevant characteristic. What matters is whether something works, i.e. succeeds. If you think that other methods are less likely to work, how are they of greater expected value? I probably parsed some of your terms incorrectly.
Whether the project reached the desired goal, versus whether that goal will actually work. If Nick and Eliezer both agreed about some design that "this is how you build a FAI", then I expect it will work. However, I don't think it's likely that would happen. It's more likely they will say "this is how you build a proper Oracle AI", but less likely the Oracle will end up being safe.
Okay, but I still don't understand how a project with lower probability of "actually working" can be of higher expected value. I'm referring to this statement:
The argument you seem to be giving in support of higher expected value of other methods is that they are "more likely to be implementable" (a project reaching its stated goal, even if that goal turns out to be no good), but I don't see how is that an interesting property.
He didn't say other architectures would be no good, he said they're less likely to be safe.
He thinks the distribution P(Outcome | do(complete Oracle AI project)) isn't as highly peaked at Weirdtopia as P(outcome | do(complete FAI)); Oracle AI puts more weight on regions like "Lifeless universe", "Eternal Torture", "Rainbows and Slow Death", and "Failed Utopia".
However, "Complete FAI" isn't an actionable procedure, so he examines the chance of completion conditional on different actions he can take. "Not worth pursuing because non-implementable" means that available FAI supporting actions don't have a reasonable chance of producing friendly AI, which discounts the peak in the conditional outcome distribution at valuable futures relative to do(complete FAI). And supposedly he has some other available oracle AI supporting strategy which fares better.
Eating a sandwich isn't as cool as building an interstellar society with wormholes for transportation, but I'm still going to make a sandwich for lunch, because it's going to work and maybe be okay-ish.