LESSWRONG
LW

zeshen — LessWrong

Replying toMy AI Predictions for 2027

My AI Predictions for 2027

Thanks for writing this up! I also want to register that I agree with all of this, maybe except for the part where AIs can't tell novel funny jokes - I expect this to be relatively easy. But of coursre it depends on the definition of 'novel'.

I struggled to do this exercise myself because when I looked at AI as a normal technology I felt like I basically agree with most of their thinking, but it was also hard to find concrete differences between their predictions and AI2027 at least in the near term. For example, for things like "LLMs are broadly acknowledged to be plateauing", it's probably going to be concurrently both true and false in a way that's hard to resolve - a lot of people may complain that it's plateauing but the benchmark scores and the usage stats could show otherwise.

Replying toOn thinking about AI risks concretely

zeshen7mo

On thinking about AI risks concretely

Yeah, at least "literally everyone dies" has a concrete ending even though it doesn't have concrete intermediate concrete steps. Gradual disempowerment seem less concrete on both the ending and the intermediate steps, so it becomes even less action-relevant.

On thinking about AI risks concretely

zeshen

7mo

Epistemic status: Shower thoughts, not meant to be rigorous.

There seems to be a fundamental difference in how I (and perhaps others as well) think about AI risks as compared to the dominant narrative on LessWrong (hereafter the “dominant narrative”), that is difficult to reconcile.

The dominant narrative is that once we have AGI, it would recursively improve itself until it becomes ASI which inevitably kills us all. To which someone like me might respond with “ok, but how exactly?”. The typical response to that might be that the “how” doesn’t matter, we all die anyway. A popular analogy is that while you don’t know how exactly Magnus Carlsen is going to beat you... (read 938 more words →)

Replying toFoom & Doom 1: “Brain in a box in a basement”

zeshen7mo

Foom & Doom 1: “Brain in a box in a basement”

…But I’m not sure that actual existing efforts towards delaying AGI are helping.

But perhaps actual existing efforts to hype up LLMs are helping? I am sympathetic to François Chollet's position:

OpenAI basically set back progress towards AGI by quite a few years probably like five to 10 years for two reasons. They caused this complete closing down of Frontier research publishing but also they triggered this initial burst of hype around LLMs and now LLMs have sucked the oxygen out of the room.

Replying to(The) Lightcone is nothing without its people: LW + Lighthaven's big fundraiser

zeshen1y

(The) Lightcone is nothing without its people: LW + Lighthaven's big fundraiser

Is there any difference between donating through Manifund or directly via Stripe?

Replying toInformation vs Assurance

zeshen1y

Information vs Assurance

This happened all the time at my line of work. Forecasts become targets and you become responsible for meeting them. So whenever I was asked to provide a forecast, I will either i) ask as many questions as I need to know the exact purpose of the request, and produce a forecast that meets exactly that intent, or ii) pick a forecast and provide it, but first list down all the assumptions and caveats behind the forecast that I can possibly think of. With time, I'd also get a sense of who I need to be extra careful with when providing any forecasts because of all sorts of ways that might backfire.

zeshen2y

Agreed. I'm also pleasantly surprised that your take isn't heavily downvoted.

-4

Replying toWe might be missing some key feature of AI takeoff; it'll probably seem like "we could've seen this coming"

zeshen2y*

We might be missing some key feature of AI takeoff; it'll probably seem like "we could've seen this coming"

There’ll be discussions about how these systems will eventually become dangerous, and safety-concerned groups might even set up testing protocols (“safety evals”).

My impression is that safety evals were deemed irrelevant because a powerful enough AGI, being deceptively aligned, would pass all of them anyway. We didn't expect the first general-ish AIs to be so dumb, like how GPT-4 was being so blatant and explicit about lying to the TaskRabbit worker.

-1

Replying toDeep Honesty

zeshen2y

Deep Honesty

Scott Alexander talked about explicit honesty (unfortunately paywalled) in contrast with radical honesty. In short, explicit honesty is being completely honest when asked, and radical honesty is being completely honest even without being asked. From what I understand from your post, it feels like deep honesty is about being completely honest about information you perceive to be relevant to the receiver, regardless of whether the information is explicitly being requested.

Scott also links to some cases where radical honesty did not work out well, like this, this, and this. I suspect deep honesty may lead to similar risks, as you have already pointed out.

And with regards to:

“what is kind, true, and useful?”

I think... (read more)

Replying toWhy is AGI/ASI Inevitable?

zeshen2y

Why is AGI/ASI Inevitable?

Can't people decide simply not to build AGI/ASI?

Yeah, many people, like the majority of users on this forum, have decided to not build AGI. On the other hand, other people have decided to build AGI and are working hard towards it.

Side note: LessWrong has a feature to post posts as Questions, you might want to use it for questions in the future.

Replying toLLMs seem (relatively) safe

zeshen2y

LLMs seem (relatively) safe

Definitely. Also, my incorrect and exaggerated model of the community is likely based on the minority who have a tendency of expressing those comments publicly, against people who might even genuinely deserve those comments.

Non-loss of control AGI-related catastrophes are out of control too

Yi-Yang

Yi-Yang, Mo Putera, zeshen

For the Open Philanthropy AI Worldview Contest

Executive Summary

Conditional on AGI being developed by 2070, we estimate the probability that humanity will suffer an existential catastrophe due to loss of control (LoC) over an AGI system at ~6%, using a quantitative scenario model that attempts to systematically assess the relative probabilities of all AGI-driven catastrophes in a collectively exhaustive manner (details here; spreadsheet here).
We also estimate the overall probability that humanity will suffer an existential catastrophe due to an AGI-related catastrophe at ~12%. The remaining 12% - 6% = 6% of the probability mass derives from non-LoC AGI catastrophes (e.g. AI-exacerbated x-risks).
Given our estimate that p(existential catastrophe | non-LoC) ≈ p(existential catastrophe | LoC), we argue that Open

... (read 7116 more words →)

A newcomer’s guide to the technical AI safety field

zeshen

This post was written during Refine. Thanks to Jonathan Low, Linda Linsefors, Koen Holtman, Aaron Scher, and Nicholas Kees Dupuis for helpful discussion and feedback.

Disclaimer: This post reflects my current understanding of the field and may not be an accurate representation of it. Feel free to comment if you feel that there are misrepresentations.

Motivations

I remember being fairly confused when I first started reading AI safety related posts, especially when it pertains to specific ideas or proposals, as there may be implicit assumptions behind those posts that relies on some background understanding about the research agenda. I have since had the opportunity to clear up many of those confusions by talking to many people... (read 2797 more words →)

Embedding safety in ML development

zeshen

This post was written as part of Refine. Thanks to Adam Shimi, Alexander Gietelink Oldenziel, and Vanessa Kosoy for helpful discussion and feedback.

Summary

This post aims to:

Advocate for embedding safety into development of machine learning models
Propose a framing on how to think about safety, where three factors contribute to an AI being dangerous, namely misalignment, optimization, and influence
Discuss the pros and cons of this framing

None of the ideas in this post are novel, and some of them may not even be practical, but they may be a useful framing to the problem of AI safety.

Introduction

Motivations

There are many ways to get to AGI. It could be recursively self improving AI, or something more brain-like, or something... (read 5130 more words →)

aisafety.community - A living document of AI safety communities

zeshen

zeshen, plex

Thanks to plex for co-authoring the post (co-authors are currently not reflected in EA Forum when crossposted from LessWrong).

*The AI Safety Communities logo, by DALL-E 2*

The AI safety field has been rapidly growing over the last few years, and more and more communities have been sprouting up all over the world, both physically and online.

The Alignment Ecosystem Development (AED) team identified having a living document listing them all as a low-hanging fruit, and volunteers are maintaining this live database of AI safety communities.

Here’s what you can do now:

Join communities you’re interested in and participate.
Link this to your friends.
Add communities to the document if they’re missing (account creation with coda.io is required)
Improve the descriptions

... (read more)

My Thoughts on the ML Safety Course

zeshen

This summary was written as part of Refine. The ML Safety Course is created by Dan Hendrycks at the Center for AI Safety. Thanks to Adam Shimi and Thomas Woodside for helpful feedback.

Overview

Background

I recently completed the ML Safety Course by watching the videos and browsing through the review questions, and subsequently writing a short summary. As an engineer in the upstream oil and gas industry with some experience in dealing with engineering safety, I find the approach of the course of thinking in this framework especially valuable.

This post is meant to be a (perhaps brutally) honest review of the course despite me having no prior working experience in ML. It may end up reflecting my ignorance of the... (read 4924 more words →)

Summary of ML Safety Course

zeshen

This summary was written as part of Refine. The ML Safety Course is created by Dan Hendrycks at the Center for AI Safety. Thanks to Linda Linsefors and Chris Scammel for helpful feedback.

Epistemic status: Low effort post intended for my own reference. Not endorsed by the course creators.

I have also written a review of the course here.

Risk Analysis

Risk Decomposition

A risk can be decomposed into its vulnerability, hazard exposure, and hazard (probability and severity). They can be defined as below, with an example in the context of the risk of contracting flu-related health complications.

Hazard: a source of danger with the potential to harm, e.g. flu prevalence and severity
Hazard exposure: extent to which elements (e.g., people, property, systems)

... (read 1647 more words →)

Levels of goals and alignment

zeshen

This post was written as part of Refine. Thanks to Adam Shimi, Lucas Teixeira, Linda Linsefors, and Jonathan Low for helpful feedback and comments.

Epistemic status: highly uncertain. This post reflects my understanding of the terminologies and may not reflect the general consensus of AI alignment researchers (if any).

Motivation

I have been very confused with the various terminologies of alignment e.g. ‘inner alignment’, ‘outer alignment’, ‘intent alignment’, etc for the longest time. For instance, someone might talk about a particular example of inner misalignment and I would be left wondering how the presence of a mesa-optimizer was established, only to find out much later that we were never on the same page at all.

Through... (read 1754 more words →)

What if we approach AI safety like a technical engineering safety problem

zeshen

This post has been written for the second Refine blog post day, at the end of the first week of iterating on ideas and concretely aiming at the alignment problem. Thanks to Adam Shimi, Paul Bricman, and Daniel Clothiaux for helpful discussion and comments.

Introduction

This post aims to provide a general introduction to technical safety used in engineering related industries, describe some commonly used tools, and attempt to draw parallels between technical safety and AI safety. I am uncertain about the value of thinking in this framework towards building safe AI, but I expect this post to at least be useful as a productive mistake.

To prevent confusion:

I use the term ‘technical safety’ to refer loosely

... (read 1943 more words →)

LESSWRONG
LW

LESSWRONG
LW

zeshen

aisafety.community - A living document of AI safety communities

I missed the crux of the alignment problem the whole time

My Thoughts on the ML Safety Course

A newcomer’s guide to the technical AI safety field

zeshen

On thinking about AI risks concretely

Non-loss of control AGI-related catastrophes are out of control too

Is there a way to sort LW search results by date posted?

A newcomer’s guide to the technical AI safety field

Embedding safety in ML development

aisafety.community - A living document of AI safety communities

My Thoughts on the ML Safety Course

zeshen

aisafety.community - A living document of AI safety communities

I missed the crux of the alignment problem the whole time

My Thoughts on the ML Safety Course

A newcomer’s guide to the technical AI safety field

zeshen

On thinking about AI risks concretely

Non-loss of control AGI-related catastrophes are out of control too

Is there a way to sort LW search results by date posted?

A newcomer’s guide to the technical AI safety field

Embedding safety in ML development

aisafety.community - A living document of AI safety communities

My Thoughts on the ML Safety Course

Executive Summary

Motivations

Summary

Introduction

Motivations

Overview

Background

Risk Analysis

Risk Decomposition

Motivation

Introduction