This is a linkpost for https://scottaaronson.blog/?p=7266
I don't think this framework is good, and overall I expected much more given the title. The name "five worlds" is associated with a seminal paper that materialized and gave names to important concepts in the latent space... and this is just a list of outcomes of AI development, with that categorization by itself providing very little insight for actual work on AI.
Repeating my comment from Shtetl-Optimized, to which they didn't reply:
It appears that you’re taking collections of worlds and categorizing them based on the “outcome” projection, labeling the categories according to what you believe is the modal representative underlying world of those categories.
By selecting the representative worlds to be “far away” from each other, it gives the impression that these categories of worlds are clearly well-separated. But, we do not have any guarantees that the outcome map is robust at all! The “decision boundary” is complex, and two worlds which are very similar (say, they differ in a single decision made by a single human somewhere) might map to very different outcomes.
The classification describes *outcomes* rather than actual worlds in which these outcomes come from.
Some classifications of the possible worlds would make sense if we could condition on those to make decisions; but this classification doesn’t provide any actionable information.
I agree that it seems like a pretty low value addition to the discourse and neither provides any additional insight, not do their categories structure the problem in a particularly helpful way. That may be exaggerated, but it feels like a plug to insert yourself into a conversation where you have nothing to contribute otherwise.
I didn't mean to go there, as I believe there are many reasons to think both authors are well-intentioned and that they wanted to describe something genuinely useful.
It's just that this contribution fails to live up to its title or to sentences like "In other words, no one has done for AI what Russell Impagliazzo did for complexity theory in 1995...". My original comment would be the same if it was an anonymous post.
Zvi discusses this in detail in Section 16, "Potential Future Scenario Naming" of his May 4, 2023 post AI #10: Code Interpreter and Geoff Hinton
I found it interesting to compare their map of AI outcomes with a very differently structured map (of 7 broad bins linearly ordered by the outcome quality) shared by Nate Soares on Oct 31, 2022 in the following post:
Superintelligent AI is necessary for an amazing future, but far from sufficient
A relatively short April 27, 2023 post attempting to classify AI outcomes into 5 main cases:
The authors elaborate:
Extensive discussion in the comments. Eliezer writes a comment, and some of the replies to that comment are quite informative.