4mo

Adversarial evaluations test whether safety measures hold when AI systems actively try to subvert them. Red teams construct attacks, blue teams build defenses, and we measure whether containment protocols survive adversarial pressure. We are building evaluations to test whether entire societal defensive processes can maintain their core functions when facing coordinated attacks from multiple agents. We think of this as building the capability to stress-test the resilience of things like democratic mechanisms, scientific consensus formation, and social dynamics under adversarial pressure. This post shows - how these evaluations can work, what we learned from some preliminary experiments, and the research agenda this opens up.

Introduction

To establish why multi-agent evaluations matter, we can walk through a concrete... (read 2674 more words →)

A Phylogeny of Agents

Jonas Hallgren

Jonas Hallgren, markov

6mo

In Douglas Hofstadter's "Gödel, Escher, Bach," he explores how simple elements give rise to complex wholes that seem to possess entirely new properties. An ant colony provides the perfect real-world example of this phenomenon - a goal directed system without much central control. This system would be considered agentic under most definitions of agency, as it is goal directed, yet what type of agent is it? Can it be multiple agents at the same time? We’ll argue that this is the case and that if we map out the ways different fields describe ant colonies we’ll be better able to solve x-risk related problems.

Introduction

Watch a harvester ant colony during its morning foraging... (read 1780 more words →)

[Paper] Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods

markov

markov, Charbel-Raphaël

9mo

"If you cannot measure it, you cannot improve it." - Lord Kelvin

The science of AI safety evaluations is still nascent, but it is making progress! We know much more today than we did two years ago.

We tried to make this knowledge accessible by writing a literature review and systematizing all the knowledge.

We wanted to go beyond a central source that just summarized collected knowledge, so we put a lot of effort into distillation, and disentangling concepts that are often presented in a messy way. We created original visualizations and gathered others from many different sources to accompany these explanations.

E.g. Disentangling truth, honesty, hallucination, deception and scheming.

The review provides a taxonomy of AI... (read more)

Replying toAI Safety 101 : Capabilities - Human Level AI, What? How? and When?

markov2y

AI Safety 101 : Capabilities - Human Level AI, What? How? and When?

Thanks for the comment! I'll add a sentence or a footnote for both loss, and weights in the sections you mentioned. As for forecasting in section 5.2, that claim is imagining something like

Different copies of a model can share parameter updates. For instance, ChatGPT could be deployed to millions of users, learn something from each interaction, and then propagate gradient updates to a central server where they are averaged together and applied to all copies of the model. (source)

This is slightly different from what is happening currently. Models are not undergoing online learning as far as I know. They are trained, they are deployed, and occasionally they are fine-tuned but that's it.... (read more)

Replying toAI Safety 101 : Capabilities - Human Level AI, What? How? and When?

markov2y

AI Safety 101 : Capabilities - Human Level AI, What? How? and When?

Fixed. Thanks :)

AI Safety 101 : Capabilities - Human Level AI, What? How? and When?

markov

markov, Charbel-Raphaël

Meta-Notes

This is a republish of a previous post, after the previous version went through heavy editing, updates and changes. The text has been expanded, content moved around/added/deleted.

Estimated reading time: 2 Hours 40 minutes reading at 100 wpm. Given the density of material covered in this chapter, if someone is encountering these arguments for the first time, I think the lesswrong projected time is too low.

Estimated reading time (including Appendices): 3 Hours 40 minutes reading at 100 wpm. The appendices are meant for people already quite familiar with topics in machine learning.

If you feel there are any mistakes or misrepresentations of anyone's views in any section please let us know.

0.0: Overview

State-of-the-Art AI. We... (read 16054 more words →)

Replying toAI Safety Chatbot

markov2y

AI Safety Chatbot

Thanks for the feedback on how to parse out feedback :)

We do have logs for everything, but as Zack pointed out we don't currently have the processes in place to automatically recover specific inputs from the logs which were meant as feedback.

AI Safety Chatbot

markov

markov, Robert Miles

Hello World! The AISafety.info team is launching a prototype of the AI Safety Chatbot.

The chatbot uses a dataset of alignment literature to answer any questions related to AI safety that you might have, while also citing established sources. Please keep in mind that this is a very early prototype and despite citing references, it may still provide inaccurate or inappropriate information.

The overall objective is to help people better understand AI Safety issues based on alignment research using an LLM. This helps with tailoring content to the user's needs and technical level. The chatbot can hopefully be used by both newcomers to AI safety, as well as researchers and engineers who want to get up to... (read 916 more words →)

Newcomers to the AI Safety arguments might be under the impression that there will be discrete cutoffs, i.e. either we have HLAI or we dont. The point of (t,n) AGI is to give a picture of what a continuous increase in capabilities looks like. It is also slightly more formal than the simple "words based" definitions of AGI. If you know of a more precise mathematical formulation of the notion of general and super intelligences, I would love if you could point me towards it so that I can include that in the post.

As for Four Background Claims, the reason for inclusion is to provide an intuition behind why general intelligence is important. And that even though future systems might be intelligent it is not the default case that they will either care about our goals, or even follow our goals in the way as intended by the designers.

I think the point of Bio Anchors was to give a big upper bound, and not say this is exactly when it will happen. At least that is how I perceive it. People who might be at a 101 level still probably have the impression that capabilities heavy AI is like multiple decades if not centuries away. The reason I have bio anchors here, is to try to point towards the fact that we have quite likely at most until 2048. Then based on that upper bound we can scale back further.

We have the recent OpenAI report that extends bio anchors - What a compute-centric framework says about takeoff speeds (https://www.openphilanthropy.org/research/what-a-compute-centric-framework-says-about-takeoff-speeds/). There is a comment under meta-notes that mentioned that I plan to include updates to timelines and takeoff in a future draft based on this report.

Replying toAI Safety 101 : Reward Misspecification

markov2y

AI Safety 101 : Reward Misspecification

Thanks for the feedback. I actually had an entire subsection in an earlier draft that covered Reward is not the optimization target. I decided to move it to the upcoming chapter 3 which covers optimization, goal misgen and inner alignment. I thought it would fit better as an intro section there since it ties the content back to the previous chapter, while also differentiating rewards from objectives. This flows well into differentiating which goals the system is actually pursuing.

AI Safety 101 : Reward Misspecification

markov

Overview

Reinforcement Learning: The chapter starts with a reminder of some reinforcement learning concepts. This includes a quick dive into the concept of rewards and reward functions. This section lays the groundwork for explaining why reward design is extremely important.
Optimization: This section briefly introduces the concept of Goodhart's Law. It provides some motivation behind understanding why rewards are difficult to specify in a way such that they do not collapse in the face of immense optimization pressure.
Reward misspecification: With a solid grasp of the notion of rewards and optimization the readers are introduced to one of the core challenges of alignment - reward misspecification. This is also known as the Outer Alignment problem.

... (read 9018 more words →)

Is AI Safety dropping the ball on privacy?

markov

TL;DR

The lack of privacy-preserving technologies facilitates better predictive models of human behavior. This accelerates several existential epistemic failure modes by enabling higher levels of deceptive and power-seeking capabilities in AI models.

What is this post about?

This post is not about things like government panopticons, hiding your information from ‘the public internet’, blurring your face on online videos, hiding from people who might look you up on Google or Facebook, or hackers getting access to your information, etc… While these issues might also be problematic, they do not pose x-risks in my mind.

This post is about things like surveillance capitalism or technofeudalism leading to an unregulatable and eventually uncontrollable Robust Agent Agnostic Process (RAAP)^[1]. This causes an... (read 1964 more words →)

Stampy's AI Safety Info - New Distillations #4 [July 2023]

markov

Hey! This is another update from the distillers at AI Safety Info.

Here are a couple of the answers that we wrote up over the last month (July 2023). As always let us know in the comments if there are any questions that you guys have that you would like to see answered.

The list below redirects to individual links, and the collective URL above renders all of the answers in the list on one page at once.

These are the new articles. There has also been significant work in overhauling already live on site articles to improve their quality based on the feedback we have been receiving from readers.

markov

Hey! This is another update from the distillers at the AI Safety Info website (and its more playful clone Stampy).

Here are a couple of the answers that we wrote up over the last month (May 2023). As always let us know in the comments if there are any questions that you guys have that you would like to see answered.

The list below redirects to individual links, and the collective URL above renders all of the answers in the list on one page at once.

How can I help tree

There are many different types of individuals from all manner of different backgrounds who want to contribute to AI Safety. So we added a FAQ with advice on... (read 300 more words →)

Stampy's AI Safety Info - New Distillations #2 [April 2023]

markov

Hey! This is another update from the distillers at the AI Safety Info website (and its more playful clone Stampy).

Here are a couple of the answers that we wrote up over the last month (April 2023). As always let us know if there are any questions that you guys have that you would like to see answered.

The list below redirects to individual links, and the collective URL above renders all of the answers in the list on one page at once.

... (read 169 more words →)

Replying toIs AI Safety dropping the ball on privacy?

markov3y

Is AI Safety dropping the ball on privacy?

TOR is way too slow and google hates serving content to TOR users. I2P might be faster than TOR but the current adoption is way too low. Additionally, it doesn't help that identity persistence is a regulatory requirement in most jurisdictions because it helps traceability against identity theft, financial theft, fraud, etc... Cookie cleaning means they have to log in every time which for most people is too annoying.

I acknowledge that there are ways to technically poison existing data. The core problem though is finding things that both normal people and also technically adept (alignment researchers/engineers/...) would actually be willing to do.

The general vibe I see right now is - * shrug... (read more)

Replying toIs AI Safety dropping the ball on privacy?

markov3y

Is AI Safety dropping the ball on privacy?

I understand your original comment a lot better now. My understanding of what you said is that open source intelligence that anyone provides through their public persona is revealing more than enough information to be damaging. The little that is sent over encrypted channels is just cherries on the cake. So the only real way to avoid manipulation is to first hope that you have not been a very engaged member of the internet for the last decade, and also primarily communicate over private channels.

I suppose I just underestimated how much people actually post stuff online publicly.

One first instinct response I had was identity isolation. That was something I was going to... (read more)

Replying toIs AI Safety dropping the ball on privacy?

markov3y

Is AI Safety dropping the ball on privacy?

I am trying to be as realistic as I can while realizing that privacy is inversely proportional to convenience.

So no, of course you should not stop making lesswrong posts.

The main things I suggested were - removing the ability to use data by favoring E2EE, and additionally removing the ability to hoard data, by favoring decentralized (or local) storage and computation.

As an example just favor E2EE services for collaborating instead of drive, dropbox, or office suite if you have the ability to do so. I agree that this doesn't solve the problem but at least it gets people accustomed to thinking about using privacy-focused alternatives. So it is one step.

Another example would be... (read more)

Replying toIs AI Safety dropping the ball on privacy?

markov3y

Is AI Safety dropping the ball on privacy?

I did consider the distinction between a model of humans vs. a model of you personally. But I can't really see any realistic way of stopping the models from having better models of humans in general over time. So yeah, I agree with you that the small pockets of sanity are currently the best we can hope for. It was mainly to spread the pocket of sanity from infosec to the alignment space is why I wrote up this post. Because I would consider the minds of alignment researchers to be critical assets.

As to why predictive models of humans in general seems unstoppable - I thought it might be too much to... (read more)

LESSWRONG
LW

LESSWRONG
LW

markov

AI Safety Chatbot

Is AI Safety dropping the ball on privacy?

AI Safety 101 : Capabilities - Human Level AI, What? How? and When?

Stampy's AI Safety Info - New Distillations #1 [March 2023]

markov

markov

System Level Safety Evaluations

A Phylogeny of Agents

[Paper] Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods

AI Safety 101 : Capabilities - Human Level AI, What? How? and When?

AI Safety Chatbot

AI Safety 101 : Reward Misspecification

Is AI Safety dropping the ball on privacy?

AI Safety 101

markov

AI Safety Chatbot

Is AI Safety dropping the ball on privacy?

AI Safety 101 : Capabilities - Human Level AI, What? How? and When?

Stampy's AI Safety Info - New Distillations #1 [March 2023]

markov

markov

System Level Safety Evaluations

A Phylogeny of Agents

[Paper] Safety by Measurement: A Systematic Literature Review of AI Safety Evaluation Methods

AI Safety 101 : Capabilities - Human Level AI, What? How? and When?

AI Safety Chatbot

AI Safety 101 : Reward Misspecification

Is AI Safety dropping the ball on privacy?

AI Safety 101

Introduction

Introduction

Meta-Notes

0.0: Overview

Overview

TL;DR

What is this post about?

How can I help tree