Jono

Jono9moQuick Take

Is everyone dropping the ball on cryonics? I'm considering career directions and my P(doom | no pause) is high and my P(doom | I work against X-risk) is close enough to my vanilla P(doom) that I wonder I should pick up this ball instead.

Replying toWinning the power to lose

Jono9mo

Winning the power to lose

Directed at the rest of the comment section: Cryogenic Suspension is an option for those who would die before the AGI launch.

If you don't like the odds that your local Suspension service preserves people well enough, then you still have the option to personally improve it before jumping to other, potentially catastrophic solutions.

The value difference commenters keep pointing out, needs to be far bigger then they represent it to be, to be relevant in a discussion on whether we should increase X-risk for some other gain.

The fact we don't live in a world where ~all accelerationalists invest in cryo suspensions, makes me think they are in fact not looking at what they're steering towards.

Replying toFake thinking and real thinking

Jono9mo

Fake thinking and real thinking

Tangentially, I fear that the model of psychology of "human agency originates from trying to correct the world such that it fits our false beliefs about it" is correct.

I hate systematically believing false things, but do notice that my times of greatest labor have a strong sense of "trying to correct the world towards how it should be", where the "should be" is epistemically flavored. Meditations that aim to grok how the world is not how it should be, seem to diminish my drive to rectify the flaw and yield acceptance instead.

Replying toFake thinking and real thinking

Jono9mo

Fake thinking and real thinking

Thank you. I want to stress (you already spent some words on this) that we don't always think with the aim to find truth, or find some particular truth.

Thinking as how to achieve some goal looks different from thinking as how to best describe some facet of the world. When evaluating your thoughts (or when someone else evaluates yours) I think it paramount to know what motivated your reasoning in the first place.

My tag — not for getting to think real, but for knowing whether your thought was good — is to understand the constraints the thought was under.
Who can learn the thought?
In what way do observers look at the thought (is... (read more)

Replying toAI Safety Memes Wiki

Jono10mo

AI Safety Memes Wiki

Very cool, I'm not seeing a table of contents on aisafety.info however.

Jono10mo

Thank you very much.
I imagined the forecasting AI to not be smart enough to be able to simulate a tiler that knows it is being simulated by us.
Perhaps that constraint is so large that the forecasts cannot be reliable.

Jono10moQuick Take

Does this help outer alignment?

Goal: tile the universe with niceness, without knowing what niceness is.

Method

We create:
- a bunch of formulations of what niceness is.
- a tiling AI, that given some description of niceness, tiles the universe with it.
- a forecasting AI, that given a formulation of niceness, a description of the tiling AI, a description of the universe and some coordinates in the universe, generates a prediction of what the part of the universe at the coordinates looks like after the tiling AI has tiled it with the formulation of niceness.

Following that, we feed our formulations of niceness into the forecasting AI, randomly sample some coordinates and evaluate whether the resulting predictions... (read more)

Replying toFinishing The SB-1047 Documentary In 6 Weeks

Jono11mo

Finishing The SB-1047 Documentary In 6 Weeks

I have looked around a bit and have not seen any updates since November, which estimated this be finished in early February.
Could you give another update, or link a more recent one if it exists?

Jono1yQuick Take

P(doom) can be approximately measured.
If reality fluid describes the territory well, we should be able to see close worlds that already died off.

For nuclear war we have some examples.
We can estimate the odds that the Cuban missile crisis and Petrov's decision went badly. If we accept that luck was a huge factor in us surviving those events (or not encountering events like it), we can see how unlikely our current world is to still live.

A high P(doom) implies that we are about to (or already did) encounter some very unlikely events that worked out suspiciously well for our survival. I don't know how public a registry of events like this should be,... (read more)

Jono's Shortform

Jono

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Replying toWhat should I do? (long term plan about starting an AI lab)

Jono2y

What should I do? (long term plan about starting an AI lab)

I don't know if you have already, but this might be the time to take a long and hard look at the probblem and consider whether deep learning is the key to solving it.

What is the problem?

reckless unilateralism? -> go work for policy or chip manufacturing
inabillity to specify human values? -> that problem looks not DL at all to me
powerful hackers stealing all the proto-AGIs in the next 4 years? -> go cybersec
deception? -> (why focus there? why make an AI that might deceive you in the first place?) but that's pretty ML, though I'm not sure interp is the way to go there
corrigibility? -> might be ML, though I'm not sure

Jono

With optimisation-robust I mean that it withstands point 27 from AGI Ruin:

When you explicitly optimize against a detector of unaligned thoughts, you're partially optimizing for more aligned thoughts, and partially optimizing for unaligned thoughts that are harder to detect. Optimizing against an interpreted thought optimizes against interpretability.

Are you aware of any person or group that is working expressly on countering this failure mode?

Closed-Source Evaluations

Jono

Public tripwires are no tripwires.

I'm writing a quick and dirty post because the alternative is that I wait for months and maybe not write it after all. I am broadly familiar with the state of interpretability research but do not know what the state of model evaluations is at the moment.

The interpretability win screen comes after the game over.

There is an interpretability-enabled win condition at the point where billion-parameter networks become transparent enough that we robustly can detect deception.
We are paperclipped long before that since their adjacant insights predictably lead to quicker iteration cycles, new architecture discoveries, insight into strategic planning and other capabilities AGI labs are looking for.

Tripwires stop working when

... (read 276 more words →)

AI demands unprecedented reliability

Jono

My AI pessimism is not addressed by optimistic arguments that do not promise the level of reliability required for a technology that our societies vital functions are expected to rely on.

Historically, useful improvements are hard to reverse and solutions can suddenly fail after we had become dependent on them. The potato seemed perfect for Ireland until they all blighted. Dutch dams seemed good, until they broke. Banks seem to keep suddenly failing on the regular.

AI is useful and we rely on it. Our culture is under heavy influence of search engines and algorithmic recommendations, people entrust their lives to self-driving cars and companies entrust sensitive or copy-righted data to generative models.

AI relies... (read 324 more words →)

LESSWRONG
LW

LESSWRONG
LW

AI demands unprecedented reliability

Closed-Source Evaluations

Is anyone developing optimisation-robust interpretability methods?

Jono's Shortform

Jono

Jono's Shortform

Is anyone developing optimisation-robust interpretability methods?

Closed-Source Evaluations

AI demands unprecedented reliability

Jono

AI demands unprecedented reliability

Closed-Source Evaluations

Is anyone developing optimisation-robust interpretability methods?

Jono's Shortform

Jono

Jono's Shortform

Is anyone developing optimisation-robust interpretability methods?

Closed-Source Evaluations

AI demands unprecedented reliability

The interpretability win screen comes after the game over.

Tripwires stop working when