NB: this post has lost footnotes. Please read the version on Effective Altruism Forum with footnotes. In our first LW post on the Gaia Network, we framed it as a solution to the challenges of building safe, transformative AI. However, the true potential of Gaia as a “world-wide web of...
Introduction The Open Agency Architecture (OAA) proposal by Davidad has been gaining traction as a conceptual architecture to allow humanity to benefit from the nearly boundless upsides of AI while rigorously limiting its risks. As quoted here, the tl;dr is to “utilise near-AGIs to build a detailed world simulation, train...
Way late to the game, but just arrived here through Richard Ngo's recent Substack post and figured I might just mention that this is identical to the Ergodicity Economics program being developed by Ole Peters and Alex Adamou since about 2011. Probably worth checking out for some cross-pollination!
Latecomer, but as this relates to some of my prior work on self- and other-modeling, I thought I'd comment... The consistently high task accuracy displayed on Figure D suggests that even your smallest neural network is significantly over-capacity/over-parameterized for the test dataset. Excess capacity seems to be the only way the model can take on the expensive self-modeling task (*) without losing accuracy on the main task. Indeed, this would suggest that the explanation for the regularization benefit of self-modeling here is precisely that it soaks up the excess capacity, avoiding overfitting. But obviously, you can have too much of a good thing -- as the experiments with fewer hidden layers show,... (read more)
@Épiphanie Gédéon this is great, very complementary/related to what we've been developing for the Gaia Network. I'm particularly thrilled to see the focus on simplicity and incrementalism, as well as the willingness to roll up one's sleeves and write code (often sorely lacking in LW). And I'm glad that you are taking the map/territory problem seriously; I wholeheartedly agree with the following: "Most safe-by-design approaches seem to rely heavily on formal proofs. While formal proofs offer hard guarantees, they are often unreliable because their model of reality needs to be extremely close to reality itself and very detailed to provide assurance."
A few additional thoughts:
NB: this post has lost footnotes. Please read the version on Effective Altruism Forum with footnotes.
In our first LW post on the Gaia Network, we framed it as a solution to the challenges of building safe, transformative AI. However, the true potential of Gaia as a “world-wide web of causal models” goes far beyond that, and in fact, justifying it in terms of its value to other use cases is key to showing its viability for AI safety. At the same time, the previous post focused more on the “what” and “why”, and didn’t really talk much about the “how”. In this piece, we’ll correct both of these flaws: we’ll visually walk... (read 4374 more words →)
Hey Steven, I'll answer your question/suggestion below. One upfront request: please let us know if this helps. We'll write a follow-up post on LW explaining this.
As mentioned in the appendix, most of what we wrote up is generalized from concrete people (not made-up, my IRL company Digital Gaia) trying to build a specific concrete AI thing (software to help farmers and leaders of regeneration projects maximize their positive environmental impact and generate more revenue by being able to transparently validate their impact to donors or carbon credit buyers). We talked extensively to people in the ag, climate and nature industries, and came to the conclusion that the lack of transparent, unbiased impact... (read 394 more words →)
The Open Agency Architecture (OAA) proposal by Davidad has been gaining traction as a conceptual architecture to allow humanity to benefit from the nearly boundless upsides of AI while rigorously limiting its risks. As quoted here, the tl;dr is to “utilise near-AGIs to build a detailed world simulation, train and formally verify within it that the AI adheres to coarse preferences and avoids catastrophic outcomes.”
We think this traction is well-deserved: the only "Safe AGI" plans that stand a chance of feasibility will operationalise outer alignment through negotiation and model-based decision-making and coordination[1]. We also agree with most of Davidad’s takes on other hard problems in AI Safety and how to tackle them with OAA.
However, if... (read 4551 more words →)
people won’t want to prioritise informationally best comments, and that their main motivation for reading comments is confirming their pre-existing worldviews. This is sort of what is customary to expect, but leaning into my optimism bias, I should plan as if this is not the case. (Otherwise, aren’t we all doomed, anyway?)
There are countermoves to this. Preferences and behaviors are malleable. There can be incentives for adopting BetterDiscourse (potentially through public good funding), peer pressure, etc.
I think this should very valuable already in this completely local regime, however, things may get even more interesting, and to recapitulate the “collaborative filtration power” of Community Notes, Pol.is, and Viewpoints.xyz, (active) users’ feedbacks are aggregated to bubble up the best comments up for new users, or for users who choose not to vote actively to tune their predictive model well. Furthermore, when users with a similar state-space already voted positively for comments that their models didn’t predict then such comments could be shown earlier to other users in the same state-space cluster, overriding the predictions of their models.
I used to think this wouldn't reach a critical mass of high-quality active... (read more)
Maybe just point to the relevant paper? https://arxiv.org/abs/2312.00752
Excellent post, a great starting point, but we must go deeper :) For instance:
To be fair, I also wake up most days and wonder how all this can be real.