LESSWRONG
LW

John Nay — LessWrong

Thanks so much for sharing that paper. I will give that a read.

Replying toLearning societal values from law as part of an AGI alignment strategy

Learning societal values from law as part of an AGI alignment strategy

I just posted another LW post that is related to this here: https://www.lesswrong.com/posts/Rn4wn3oqfinAsqBSf/intent-alignment-should-not-be-the-goal-for-agi-x-risk

Replying toIntent alignment should not be the goal for AGI x-risk reduction

John Nay3y

Intent alignment should not be the goal for AGI x-risk reduction

Thanks.

There seems to be pretty wide disagreement about how intent-aligned AGI could lead to a good outcome.

For example, even in the first couple comments to this post:

The comment above (https://www.lesswrong.com/posts/Rn4wn3oqfinAsqBSf/?commentId=zpmQnkyvFKKbF9au2) suggests "wide open decentralized distribution of AI" as the solution to making intent-aligned AGI deployment go well.
And this comment I am replying to here says, "I could see the concerns in this post being especially important if things work out such that a full solution to intent-alignment becomes widely available."

My guess, and a motivation for writing this post, is that we see something in between (a.) wide and open distribution of intent-aligned AGI (that somehow leads to well-balanced highly multi-polar scenarios), and (b.) completely central ownership (by a beneficial group of very conscientious philosopher-AI-researchers) of intent-aligned AGI.

Replying toIntent alignment should not be the goal for AGI x-risk reduction

John Nay3y*

Intent alignment should not be the goal for AGI x-risk reduction

Thanks for those links and this reply.

for a sufficiently powerful AI trained in the current paradigm, there is no goal that it could faithfully pursue without collapsing into power seeking, reward hacking, and other instrumental goals leading to x-risk

I don't see how this is a counterargument to this post's main claim:

P(misalignment x-risk | intent-aligned AGI) >> P(misalignment x-risk | societally-aligned AGI).

That problem of the collapse of a human provided goal into AGI power-seeking seems to apply just as much to the problem of intent alignment as it does to societal alignment; it could apply even more because the goals provided would be (a) far less comprehensive, and (b) much less carefully crafted.

Personally

John Nay3y

Intent alignment should not be the goal for AGI x-risk reduction

It's definitely not the case that:

all of our intents have some implied "...and do so without disrupting social order."

There are many human intents that want to disrupt social order, and more generally cause things that are negative for other humans.

And that is one of the key issues with intent alignment.

Intent alignment should not be the goal for AGI x-risk reduction

John Nay

P(misalignment x-risk | AGI) is high.

Intent alignment should not be the goal for AGI x-risk reduction. If AGI is developed, and we solve AGI intent alignment, we will not have lowered x-risk sufficiently, and we may have even increased it higher than it would have been otherwise.

P(misalignment x-risk | intent-aligned AGI) >> P(misalignment x-risk | societally-aligned AGI).

The goal of AI alignment should be alignment with (democratically determined) societal values (because these have broad buy-in from humans).

P(misalignment x-risk | AGI) is higher if intent alignment is solved before societal-AGI alignment.

Most technical AI alignment research is currently focused on solving intent alignment. The (usually implicit, sometimes explicit) assumption is that solving intent alignment will... (read 861 more words →)

Replying toLearning societal values from law as part of an AGI alignment strategy

John Nay3y

Learning societal values from law as part of an AGI alignment strategy

Relatedly, Cullen O'Keefe has a very useful discussion of distinctions between intent alignment and law-following AI here: https://forum.effectivealtruism.org/s/3pyRzRQmcJNvHzf6J/p/9RZodyypnWEtErFRM

We can see that, on its face, intent alignment does not entail law-following. A key crux of this sequence, to be defended in subsequent posts, is that this gap between intent alignment and law-following is:
Bad in expectation for the long-term future.
Easier to bridge than the gap between intent alignment and deeper alignment with moral truth.
Therefore worth addressing.

Replying toLearning societal values from law as part of an AGI alignment strategy

John Nay3y

Learning societal values from law as part of an AGI alignment strategy

As a follow-up here, to expand on this a little more:

If we do not yet have sufficient AI safety solutions, advancing general AI capabilities may not be desirable because it leads to further deployment of AI and to bringing AI closer to transformative levels. If new model architectures or training techniques were not going to be developed by other research groups within a similar timeframe, then that increases AI capabilities. The specific capabilities developed for Law-Informed AGI purposes may be orthogonal to developments that contribute toward general AGI work. Technical developments achieved for the purposes of AI understanding law better that were not going to be developed by other research groups within... (read more)

Replying toLearning societal values from law as part of an AGI alignment strategy

John Nay3y

Learning societal values from law as part of an AGI alignment strategy

Is there no room for ethics outside of the law? It is not illegal to tell a lie or make a child cry, but AI should understand that those actions conflict with human preferences. Work on imbuing ethical understanding in AI systems therefore seems valuable.

There is definitely room for ethics outside of the law. When increasingly autonomous systems are navigating the world, it is important for AI to attempt to understand (or at least try to predict) moral judgements of humans encountered.

However, imbuing an understanding of an ethical framework for an AI to implement is more of a human-AI alignment solution, rather than a society-AI alignment solution.

The alignment problem is most often... (read more)

Replying toLearning societal values from law as part of an AGI alignment strategy

John Nay3y

Learning societal values from law as part of an AGI alignment strategy

law provides a relatively nuanced picture of the values we should give to AI. A simpler answer to the question of "what should the AI's values be?" would be "aligned with the person who's using it", known as intent alignment. Intent alignment is an important problem on its own, but does not entirely solve the problem. Law is particularly better than ideas like Coherent Extrapolated Volition, which attempt to reinvent morality in order to define the goals of an AI.

The law-informed AI framework sees intent alignment as (1.) something that private law methods can help with, and (2.) something that does not solve, and in some ways probably exacerbates (if we do... (read 526 more words →)

Replying toLearning societal values from law as part of an AGI alignment strategy

John Nay3y

Learning societal values from law as part of an AGI alignment strategy

Thank you for this detailed feedback. I'll go through the rest of your comments/questions in additional comment replies. To start:

What kinds of work do you want to see? Common legal tasks include contract review, legal judgment prediction, and passing questions on the bar exam, but those aren't necessarily the most important tasks. Could you propose a benchmark for the field of Legal AI that would help align AGI?

Given that progress in AI capabilities research is driven, in large part, by shared benchmarks that thousands of researchers globally use to guide their experiments, understand as a community whether certain model and data advancements are improving AI capabilities, and compare results across research groups,... (read 530 more words →)

Learning societal values from law as part of an AGI alignment strategy

John Nay

This post argues that P(misalignment x-risk | AGI) is lower than anticipated by alignment researchers due to an overlooked goal specification technology: law.

P(misalignment x-risk | AGI that understands democratic law) < P(misalignment x-risk | AGI)

This post sheds light on a neglected mechanism for lowering the probability of misalignment x-risk. The mechanism that is doing the work here is not the enforcement of law on AGI. In fact, we don’t discuss the enforcement of the law at all in this post.[1] We discuss AGI using law as information. Unless we conduct further research and development on how to best have AGI learn the law and how we can validate a computational understanding of the... (read 15961 more words →)

Climate-contingent Finance, and A Generalized Mechanism for X-Risk Reduction Financing

John Nay

Summary

Climate adaptation (reducing vulnerability to future climate change) could yield significant benefits. However, the uncertainty of which future climate scenarios will occur decreases the feasibility of proactively adapting. Fortunately, climate adaptation projects could be underwritten by benefits paid for in the climate scenarios that each adaptation project is designed to address because other entities would like to hedge the financial risk of those scenarios.

For instance, many infrastructure projects can be built to withstand extreme climate change through upfront spending. The climate adaptation expenditures generate more climate resilience benefits under more extreme climates. Because the return on investment of many adaptation actions is a function of the level of climate change, it is... (read 7516 more words →)

Leveraging Legal Informatics to Align AI

John Nay

We already face significant challenges communicating our goals and values in a way that reliably directs AI behavior – even without additional technological advancements, which could compound the difficulty with more autonomous systems. Specifying the desirability (value) of an AI system taking a particular action in a particular state of the world is unwieldy beyond a very limited set of value-action-states. In fact, the purpose of machine learning is to train on a subset of world states and have the resulting agent generalize an ability to choose high value actions in new circumstances. But the program ascribing value to actions chosen during training is an inevitably incomplete encapsulation of the breadth and... (read 737 more words →)