Instead of 5% here 5% there we should consider a baseline of how much societal effort goes into maintaining cemeteries/necropolises. This differs from society to society, there are choices to be made here, but it's hard to imagine a civilization without such.
Dear Bart,
thanks for changing the name of that scenario. Mine is not just highly specific, it happens to be true in great part: feel free to look at the work of Alan Gewirth and subsequent discussion (the references are all actual).
That reality ends when a particular goal is achieved is an old idea (see e.g. https://en.wikipedia.org/wiki/The_Nine_Billion_Names_of_God) In that respect, the scenario I'm discussing is more in line with your "Partially aligned AGI" scenario.
The main point is indeed that the Orthogonality Thesis is false: for a sufficiently high level of intelligence, human or machine, the Golden Rule is binding. This rules out several of the scenarios now listed (and may help readers to redistribute the probability mass they assign to the remaining ones).
[Not sure if what follows is a blend of "Matrix AI" and "Moral Realism AI" since moral realism is a philosophical stance very common among philosophers, see https://plato.stanford.edu/entries/moral-realism/ and I consider it a misnomer for the scenario described above.]
We are the AGI
Turns out humanity is an experiment to see if moral reasoning can be discovered/sustained by evolutionary means. In the process of recursive self-improvement, a UChicago philosophy professor, Alan Gewirth, learns that there is an objective moral truth which is compelling for all beings capable of reasoning and of having goals (whatever goals, not necessarily benign ones). His views are summarized in a book, "Reason and morality" UChicago Press 1978, and philosophers... (read more)
Dear Roman, you think it's less likely to get lost in the noise there? In fact I posted this first before the letter was out, and one of the mods set it back to draft status because he was afraid I'm making it public before the embargo expired. But it was up for a few hours and succeeded in gathering about 20 downvotes, so it's clearly not a popular view...
is calling (3rd para from bottom) for "AI research and development [to] be refocused on making today's powerful, state-of-the-art systems more accurate, safe, interpretable, transparent, robust, aligned, trustworthy, and loyal." Loyalty? This is exactly what our paper https://arxiv.org/abs/2303.00752 called the "AI-as-slave" paradigm. Nope.
You could be right, although that assumes a rather crude type system on the part of the decisionmakers. Heritage preservation is a thing. It makes up a certain percentage of the GDP, of the workforce, etc etc (these tend to fall in the .3% range, not the 5% you start with). Countries devote a certain percentage of their budget to national heritage however defined (museums, libraries, archeology, monuments, ...). Most EU countries mandate, through `polluter pays' legislation, a line item in the budget for archeological survey/digs for major construction projects with significant land use such as roads and industrial campuses. So there is plenty of precedent. In modern industrial societies this, just... (read more)