[Sorry about the lengthiness of this post. I recommend not fixating too much on all the specific numbers and the formal apparatus. Originally the plan was to also analyze optimal timing from an impersonal (xrisk-minimization) perspective; but to prevent the text from ballooning even more, that topic was set aside...
Preamble for LessWrong I've seen many prescriptive contributions to AGI governance take the form of proposals for some radically new structure. Some call for a Manhattan project, others for the creation of a new international organization, etc. The OGI model, instead, is basically the status quo. More precisely, it is...
Yes the post explicitly considers things only from a mundane person-affecting stance, and I would not argue that this is the correct stance or the one I would all-things-considered endorse. However, it may be a component in a more plausible complex view, or a caucus in a 'moral parliament'; so I think it is worth investigating what it implies. If ethics is complicated, we may need a divide and conquer strategy, where we isolate and analyze one element at a time.
... (read 382 more words →)Existing ordinary people usually have preferences and values not captured by the QALY calculation, such as wanting their kids having happy lives, or wanting the world not to end, or even wanting
[Sorry about the lengthiness of this post. I recommend not fixating too much on all the specific numbers and the formal apparatus. Originally the plan was to also analyze optimal timing from an impersonal (xrisk-minimization) perspective; but to prevent the text from ballooning even more, that topic was set aside for future work (which might never get done). But I should at least emphasize that there are other important factors, not covered here, that would need to be taken into account if one wishes to determine which timeline would be best all things considered.]
[Working paper.[1] Version 1.0. Canonical link to future revised version of this paper.]
Developing superintelligence is not like playing Russian... (read 21481 more words →)
I'm not recommending it (haven't done a cost-benefit analysis) but Anna was considering flying less because of the fume risk, and bringing a gas mask in the carry-on seems less burdensome than refraining from flying.
Could bring a gas mask?
My biggest problem with this proposal is that it restricts AGI projects to a single entity, which I think is pretty far from the status quo.
It doesn't, though. The paper talks about OGI-N as well as OGI-1. In the former version, there are multiple AGI-developing entities.
I think that's an overstatement. There is still plenty of demand from international investors for U.S. assets, and e.g. yields on U.S. 30-year treasury bonds are not that high by historical standards (although they've been climbing somewhat since their historic lows in 2020.) If there is a reduction in confidence in other sorts of USG commitments, but maintained confidence in it's basic financial commitments and in U.S. property law, then that might support the paper's contention that the latter kind form a comparatively more reliable basis for positive-sum deals in relation to AGI-development than other types of agreements.
"I think your life expectancy and that of your loved ones (at least from a mundane perspective) is longer if AGI is developed than if it isn't."
You must have extreme confidence about this, or else your attitude about AGI would be grossly cavalier.
Regarding attitudes about AGI, that's probably a bigger topic for another time. But regarding your and your loved ones' life expectancy, from a mundane perspective (which leaves out much that is actually very relevant), it would presumably be some small number of decades without AGI---less if the people you love are elderly or seriously ill. Given aligned AGI, it could be extremely long (and immensely better in quality). So even... (read more)
Every known plan for a post-AGI world is one which I do not expect my loved ones to survive.
I think your life expectancy and that of your loved ones (at least from a mundane perspective) is longer if AGI is developed than if it isn't.
Btw, the OGI model is not primarily intended for a post-AGI world, but rather for a near-term or intermediary stage.
However, I agree that if somebody thinks that we should completely stop AGI then the OGI model would presumably not be the way to go. It is presented as an alternative to other governance models for the development of AGI (such as Manhattan project, CERN, Intelsat, etc.). This paper... (read more)
It becomes partly terminological, but I would say that China-OGI-1 would by definition be a situation in which global investors in a Chinese company that develops AGI enjoy reasonably reliable assurances that their property rights would be largely respected. It seems maybe more attractive than the closest alternatives (i.e. a situation in which AGI is developed by a Chinese company and international investors don't have reasonable assurances that their rights would be protected, or a situation in which AGI is developed by a Chinese Manhattan project)? So the factors you point to don't affect the desirability of China-OGI-1 but rather the probability of that version of the OGI model becoming instantiated.
Btw, I... (read more)
I always thought it was totally crazy for people to lump Nick Bostrom and Marc Andreessen together into TESCREAL and criticize them in the same breath, but this post plays right into such criticism.
I'm also bald...
I've seen many prescriptive contributions to AGI governance take the form of proposals for some radically new structure. Some call for a Manhattan project, others for the creation of a new international organization, etc. The OGI model, instead, is basically the status quo. More precisely, it is a model to which the status quo is an imperfect and partial approximation.
It seems to me that this model has a bunch of attractive properties. That said, I'm not putting it forward because I have a very high level of conviction in it, but because it seems useful to have it explicitly developed as an option so that it can be compared with... (read 11564 more words →)
More up front than in the title?
Societal decision-making typically uses a far narrower basis than the general person-affecting stance that this paper analyzes. For example, not only do voters and governments usually not place much weight on not-yet-conceived people that might come into existence in future millennia, but they care relatively little about what happens to currently existing persons in other countries.