[Sorry about the lengthiness of this post. I recommend not fixating too much on all the specific numbers and the formal apparatus. Originally the plan was to also analyze optimal timing from an impersonal (xrisk-minimization) perspective; but to prevent the text from ballooning even more, that topic was set aside...
Preamble for LessWrong I've seen many prescriptive contributions to AGI governance take the form of proposals for some radically new structure. Some call for a Manhattan project, others for the creation of a new international organization, etc. The OGI model, instead, is basically the status quo. More precisely, it is...
On the substance-level, btw, one "arcane consideration" that I would put a lot of weight on, even on person-affecting views, is stuff like what Bostrom talks about in the Cosmic Host paper. ... Even if we can't align AIs to human values, I feel like we at least have a duty to make them good at the building of peaceful coalitions/it being an okay thing to add to the cosmic host.
Yes, although, as that paper discusses, speed may also be important insofar as it reduces the risk of us failing to add anything at all, since that's also something the cosmic host may care about - the risk that we fail ever to produce superintelligence. (My views about those things are quite tentative, and they fall squarely into the 'arcane'. I agree on their importance.)
The average planning horizon for climate change regulation is many decades in the future. Nuclear waste management policies are expected to contain waste for hundreds of years. ... Maybe you have some other evidence that convinced you that countries and policy-makers operate on person-affecting views?
Of course they don't consistently operate on any specific moral view. But I would claim that they are less badly approximated by 'benefit currently existing citizens' than 'neutrally benefit all possible future people (or citizens) that might be brought into existence over future eons'. Much less is spent on things like nuclear waste management and preventing climate change than on providing amenities for the current population. In fact,... (read more)
Like, I think a persuasive or reasonable paper would have put its central load-bearing assumptions up front.
More up front than in the title?
it obviously shouldn't be the basis of societal decision-making, and luckily also isn't".
Societal decision-making typically uses a far narrower basis than the general person-affecting stance that this paper analyzes. For example, not only do voters and governments usually not place much weight on not-yet-conceived people that might come into existence in future millennia, but they care relatively little about what happens to currently existing persons in other countries.
Yes the post explicitly considers things only from a mundane person-affecting stance, and I would not argue that this is the correct stance or the one I would all-things-considered endorse. However, it may be a component in a more plausible complex view, or a caucus in a 'moral parliament'; so I think it is worth investigating what it implies. If ethics is complicated, we may need a divide and conquer strategy, where we isolate and analyze one element at a time.
... (read 382 more words →)Existing ordinary people usually have preferences and values not captured by the QALY calculation, such as wanting their kids having happy lives, or wanting the world not to end, or even wanting
[Sorry about the lengthiness of this post. I recommend not fixating too much on all the specific numbers and the formal apparatus. Originally the plan was to also analyze optimal timing from an impersonal (xrisk-minimization) perspective; but to prevent the text from ballooning even more, that topic was set aside for future work (which might never get done). But I should at least emphasize that there are other important factors, not covered here, that would need to be taken into account if one wishes to determine which timeline would be best all things considered.]
[Working paper.[1] Version 1.0. Canonical link to future revised version of this paper.]
Developing superintelligence is not like playing Russian... (read 21481 more words →)
I'm not recommending it (haven't done a cost-benefit analysis) but Anna was considering flying less because of the fume risk, and bringing a gas mask in the carry-on seems less burdensome than refraining from flying.
Could bring a gas mask?
My biggest problem with this proposal is that it restricts AGI projects to a single entity, which I think is pretty far from the status quo.
It doesn't, though. The paper talks about OGI-N as well as OGI-1. In the former version, there are multiple AGI-developing entities.
I think that's an overstatement. There is still plenty of demand from international investors for U.S. assets, and e.g. yields on U.S. 30-year treasury bonds are not that high by historical standards (although they've been climbing somewhat since their historic lows in 2020.) If there is a reduction in confidence in other sorts of USG commitments, but maintained confidence in it's basic financial commitments and in U.S. property law, then that might support the paper's contention that the latter kind form a comparatively more reliable basis for positive-sum deals in relation to AGI-development than other types of agreements.
"I think your life expectancy and that of your loved ones (at least from a mundane perspective) is longer if AGI is developed than if it isn't."
You must have extreme confidence about this, or else your attitude about AGI would be grossly cavalier.
Regarding attitudes about AGI, that's probably a bigger topic for another time. But regarding your and your loved ones' life expectancy, from a mundane perspective (which leaves out much that is actually very relevant), it would presumably be some small number of decades without AGI---less if the people you love are elderly or seriously ill. Given aligned AGI, it could be extremely long (and immensely better in quality). So even... (read more)
I've seen many prescriptive contributions to AGI governance take the form of proposals for some radically new structure. Some call for a Manhattan project, others for the creation of a new international organization, etc. The OGI model, instead, is basically the status quo. More precisely, it is a model to which the status quo is an imperfect and partial approximation.
It seems to me that this model has a bunch of attractive properties. That said, I'm not putting it forward because I have a very high level of conviction in it, but because it seems useful to have it explicitly developed as an option so that it can be compared with... (read 11564 more words →)
In other papers (e.g. Existential Risks (2001), Astronomical Waste (2003), and Existential Risk Prevention as a Global Priority (2013)) I focus mostly on what follows from a mundane impersonal perspective. Since that perspective is even further out of step with how humanity makes governance decisions, is it your opinion that those paper should likewise be castigated? (Some people who hate longtermism have done so, quite vehemently.) But my view is that there can be value in working out what follows from various possible theoretical positions, especially ones that have a distinguished pedigree and are taken seriously in the intellectual tradition. Certainly this is a very standard thing to do in academic philosophy, and I think it's usually a healthy practice.