LESSWRONG
LW

0
0

All of kornai's Comments + Replies

60+ Possible Futures
kornai
10

Dear Bart,

thanks for changing the name of that scenario. Mine is not just highly specific, it happens to be true in great part: feel free to look at the work of Alan Gewirth and subsequent discussion (the references are all actual). 

That reality ends when a particular goal is achieved is an old idea (see e.g. https://en.wikipedia.org/wiki/The_Nine_Billion_Names_of_God) In that respect, the scenario I'm discussing is more in line with your "Partially aligned AGI" scenario. 

The main point is indeed that the Orthogonality Thesis is false: for a suff... (read more)

Reply
60+ Possible Futures
kornai
40

[Not sure if what follows is a blend of "Matrix AI" and "Moral Realism AI" since moral realism is a philosophical stance very common among philosophers, see https://plato.stanford.edu/entries/moral-realism/ and I consider it a misnomer for the scenario described above.]

We are the AGI 

Turns out humanity is an experiment to see if moral reasoning can be discovered/sustained by evolutionary means. In the process of recursive self-improvement, a UChicago philosophy professor, Alan Gewirth, learns that there is an objective moral truth which is compelling ... (read more)

Reply
1Bart Bussmann
I agree that "Moral Realism AI" was a bit of a misnomer and I've changed it to "Convergent Morality AI". Your scenario seems highly specific. Could you try to rephrase it in about three sentences, as in the other scenarios?  I'm a bit wary about adding a lot of future scenarios that are outside of our reality and want the scenarios to focus on the future of our universe. However, I do think there is space for a scenario where our reality ends as it has achieved its goals (as in your scenario, I think?).
The open letter
kornai
50

Dear Roman, you think it's less likely to get lost in the noise there?  In fact I posted this first before the letter was out, and one of the mods set it back to draft status because he was afraid I'm making it public before the embargo expired. But it was up for a few hours and succeeded in gathering about 20 downvotes, so it's clearly not a popular view...

Reply