Let me know when you can receive donations via a UK charity.

Replying toIs there any rigorous work on using anthropic uncertainty to prevent situational awareness / deception?

Ramana KumarSep 26, 2024

Is there any rigorous work on using anthropic uncertainty to prevent situational awareness / deception?

Vaguely related perhaps is the work on Decoupled Approval: https://arxiv.org/abs/2011.08827

Dialogue on What It Means For Something to Have A Function/Purpose

johnswentworth, Ramana Kumar, Steve Petersen

Context for LW audience: Ramana, Steve and John regularly talk about stuff in the general cluster of agency, abstraction, optimization, compression, purpose, representation, etc. We decided to write down some of our discussion and post it here. This is a snapshot of us figuring stuff out together.

Hooks from Ramana:

Where does normativity come from?
Two senses of "why" (from Dennett): How come? vs What for? (The latter is more sophisticated, and less resilient. Does it supervene on the former?)
An optimisation process is something that produces/selects things according to some criterion. The products of an optimisation process will have some properties related to the optimisation criterion, depending on how good the process is at finding

... (read 4798 more words →)

Replying toConsent across power differentials

Ramana Kumar2y

Consent across power differentials

Thanks for this! I think the categories of morality is a useful framework. I am very wary of the judgement that care-morality is appropriate for less capable subjects - basically because of paternalism.

Replying toConsent across power differentials

Ramana Kumar2y

Consent across power differentials

Just to confirm that this is a great example and wasn't deliberately left out.

Consent across power differentials

Ramana Kumar

I'd like to put forward another description of a basic issue that's been around for a while. I don't know if there's been significant progress on a solution, and would be happy to pointed to any such progress. I've opted to go for a relatively rough and quick post that doesn't dive too hard into the details, to avoid losing the thought at all. I may be up for exploring details further in comments or follow-ups.

The Question: How do you respect the wishes (or preferences) of a subject over whom you have a lot of control?

The core problem: any indicator/requirement/metric about respecting their wishes is one you can manipulate (even inadvertently).

For example,... (read 657 more words →)

Replying toWithout specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

Ramana Kumar2yReview for 2022 Review

Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover

I found this post to be a clear and reasonable-sounding articulation of one of the main arguments for there being catastrophic risk from AI development. It helped me with my own thinking to an extent. I think it has a lot of shareability value.

I think this is basically correct and I'm glad to see someone saying it clearly.

Replying toSystems that cannot be unsafe cannot be safe

Ramana Kumar3y

Systems that cannot be unsafe cannot be safe

I agree with this post. However, I think it's common amongst ML enthusiasts to eschew specification and defer to statistics on everything. (Or datapoints trying to capture an "I know it when I see it" "specification".)

Replying toWhy do we care about agency for alignment?

Ramana KumarApr 23, 2023

Why do we care about agency for alignment?

This is one of the answers: https://www.alignmentforum.org/posts/FWvzwCDRgcjb9sigb/why-agent-foundations-an-overly-abstract-explanation

Replying toTeleosemantics!

Ramana Kumar3y

Teleosemantics!

The trick is that for some of the optimisations, a mind is not necessary. There is a sense perhaps in which the whole history of the universe (or life on earth, or evolution, or whatever is appropriate) will become implicated for some questions, though.

Replying toAI and Evolution

Ramana Kumar3y

AI and Evolution

I think https://www.alignmentforum.org/posts/TATWqHvxKEpL34yKz/intelligence-or-evolution is somewhat related in case you haven't seen it.

Refining the Sharp Left Turn threat model, part 2: applying alignment techniques

Vika

Vika, Vikrant Varma, Ramana Kumar, Rohin Shah

A Sharp Left Turn (SLT) is a possible rapid increase in AI system capabilities (such as planning and world modeling). This post will outline our current understanding of the most promising plan for getting through an SLT and how it could fail (conditional on an SLT occurring).

In a previous post, we broke down the SLT threat model into 3 claims:

Capabilities will generalize far (i.e. to many domains)
Alignment techniques that worked previously will fail during this transition
Humans can’t intervene to prevent or align this transition

We proposed some possible mechanisms for Claim 1, while this post will investigate possible arguments and mechanisms for Claim 2.

Plan: we use alignment techniques to find a goal-aligned model before SLT

... (read 1755 more words →)

Threat Model Literature Review

zac_kenton

zac_kenton, Rohin Shah, David Lindner, Vikrant Varma, Vika, Mary Phuong, Ramana Kumar, Elliot Catt

TL;DR: This post provides a literature review of some threat models of how misaligned AI can lead to existential catastrophe. See our accompanying post for high-level discussion, a categorization and our consensus threat model.

Where available we cribbed from the summary in the Alignment Newsletter.

For other people's overviews of some threat models, see here and here.

[ETA: When considering strengths and weaknesses of each threat model, this was done with respect to our goal of better generation/prioritization among alignment research projects. They shouldn't necessarily be read as an all-things-considered review of that work.]

Categorization

Threat Models

Goal misgeneralization + Specification Gaming ➜ MAPS

Is Power-Seeking AI an Existential Risk?

... (read 7259 more words →)

Clarifying AI X-risk

zac_kenton

zac_kenton, Rohin Shah, David Lindner, Vikrant Varma, Vika, Mary Phuong, Ramana Kumar, Elliot Catt

TL;DR: We give a threat model literature review, propose a categorization and describe a consensus threat model from some of DeepMind's AGI safety team. See our post for the detailed literature review.

The DeepMind AGI Safety team has been working to understand the space of threat models for existential risk (X-risk) from misaligned AI. This post summarizes our findings. Our aim was to clarify the case for X-risk to enable better research project generation and prioritization.

First, we conducted a literature review of existing threat models, discussed their strengths/weaknesses and then formed a categorization based on the technical cause of X-risk and the path that leads to X-risk. Next we tried to find consensus... (read 1096 more words →)

127

Autonomy as taking responsibility for reference maintenance

Ramana Kumar

I think semantics – specifically, maintaining reference relationships – is a core component of intelligent behaviour. Consequently, I think a better understanding of semantics would enable a better understanding of what machine intelligence that is “trying to do the right thing” ought to look like and how to build it. Unfortunately I’ve not managed to make these ideas particularly clear (either to myself or others). In the spirit of palimpsest-shaped research I’m sharing a draft in the hope of provoking useful thoughts.

Summary

Particular events can have meaningfulness or significance, and can carry multiple meanings at once. For example, the motion of a dial in a meter might mean that there is electricity flowing through

... (read 1207 more words →)

Refining the Sharp Left Turn threat model, part 1: claims and mechanisms

Vika

Vika, Vikrant Varma, Ramana Kumar, Mary Phuong

This is our current distillation of the sharp left turn threat model and an attempt to make it more concrete. We will discuss our understanding of the claims made in this threat model, and propose some mechanisms for how a sharp left turn could happen. This is a work in progress, and we welcome feedback and corrections.

What are the main claims of the “sharp left turn” threat model?

Claim 1. Capabilities will generalize far (i.e., to many domains)

There is an AI system that:

Performs well: it can accomplish impressive feats, or achieve high scores on valuable metrics.
Generalizes, i.e., performs well in new domains, which were not optimized for during training, with no domain-specific tuning.

Generalization is... (read 774 more words →)

Will Capabilities Generalise More?

Ramana Kumar

Nate and Eliezer (Lethality 21) claim that capabilities generalise further than alignment once capabilities start generalising far at all. However, they have not articulated particularly detailed arguments for why this is the case. In this post I collect the arguments for and against the position I have been able to find or generate, and develop them (with a few hours’ effort). I invite you to join me in better understanding this claim and its veracity by contributing your own arguments and improving mine.

Thanks to these people for their help with writing and/or contributing arguments: Vikrant Varma, Vika Krakovna, Mary Phuong, Rory Grieg, Tim Genewein, Rohin Shah.

For:

1. Capabilities have much shorter description length than

... (read 1064 more words →)

133

ELK contest submission: route understanding through the human ontology

Vika

Vika, Ramana Kumar, Vikrant Varma

This is an ELK proposal that received a prize in the category "Train a reporter that is useful to an auxiliary AI". It was coauthored with Vikrant Varma and Ramana Kumar and advised by Rohin Shah.

Training strategy and why it might work

We use an architecture with a knowledge bottleneck to train the model, consisting of an $Obs \to (Q \to A)$ function (the reporter) and a $(Q \to A) \to Prediction$ function (the predictor). The key idea is that the model has to rely on the output of the reporter to make predictions. If the reporter implements the human simulator, then it loses more information about its world model than if it implemented the direct translator (which by assumption is a lossless projection of... (read 338 more words →)

P₂B: Plan to P₂B Better

Ramana Kumar

Ramana Kumar, Daniel Kokotajlo

tl;dr: Most good plans involve taking steps to make better plans. Making better plans is the convergent instrumental goal, of which all familiar convergent instrumental goals are an instance. This is key to understanding what agency is and why it is powerful.

Planning means using a world model to predict the consequences of various courses of actions one could take, and taking actions that have good predicted consequences. (We think of this with the handle “doing things for reasons,” though we acknowledge this may be an idiosyncratic use of “reasons.”)

We take “planning” to include things that are relevantly similar to this procedure, such as following a bag of heuristics that approximates it. We’re... (read 1601 more words →)

LESSWRONG
LW

LESSWRONG
LW

Ramana Kumar

Will Capabilities Generalise More?

Clarifying AI X-risk

Thoughts on Human Models

Refining the Sharp Left Turn threat model, part 1: claims and mechanisms

Ramana Kumar

Dialogue on What It Means For Something to Have A Function/Purpose

Consent across power differentials

Refining the Sharp Left Turn threat model, part 2: applying alignment techniques

Threat Model Literature Review

Clarifying AI X-risk

Autonomy as taking responsibility for reference maintenance

Refining the Sharp Left Turn threat model, part 1: claims and mechanisms

Ramana Kumar

Will Capabilities Generalise More?

Clarifying AI X-risk

Thoughts on Human Models

Refining the Sharp Left Turn threat model, part 1: claims and mechanisms

Ramana Kumar

Dialogue on What It Means For Something to Have A Function/Purpose

Consent across power differentials

Refining the Sharp Left Turn threat model, part 2: applying alignment techniques

Threat Model Literature Review

Clarifying AI X-risk

Autonomy as taking responsibility for reference maintenance

Refining the Sharp Left Turn threat model, part 1: claims and mechanisms

Plan: we use alignment techniques to find a goal-aligned model before SLT

Categorization

Threat Models

Goal misgeneralization + Specification Gaming ➜ MAPS

Is Power-Seeking AI an Existential Risk?

Summary

What are the main claims of the “sharp left turn” threat model?

For:

1. Capabilities have much shorter description length than

Training strategy and why it might work