All of Charlie Steiner's Comments + Replies

If God has ordained some "true values," and we're just trying to find out what pattern has received that blessing, then yes this is totally possible, God can ordain values that have their most natural description at any level He wants.

On the other hand, if we're trying to find good generalizations of the way we use the notion of "our values" in everyday life, then no, we should be really confident that generalizations that have simple descriptions in terms of chemistry are not going to be good.

The second thing impacts the first thing :) If a lot of scheming is due to poor reward structure, and we should work on better reward structure, then we should work on scheming prevention.

Very interesting!

It would be interesting to know what the original reward models would say here - does the "screaming" score well according to the model of what humans would reward (or what human demonstrations would contain, depending on type of reward model)?

My suspicion is that the model has learned that apologizing, expressing distress etc after making a mistake is useful for getting reward. And also that you are doing some cherrypicking.

At the risk of making people do more morally grey things, have you considered doing a similar experiment with models... (read more)

I'm a big fan! Any thoughs on how to incorporate different sorts of reflective data, e.g. different measures of how people think mediation "should" go?

Well, I'm disappointed.

Everything about misuse risks and going faster to Beat China, nothing about accident/systematic risks. I guess "testing for national security capabilities" is probably in practice code for "some people will still be allowed to do AI alignment work," but that's not enough.

I really would have hoped Anthropic could be realistic and say "This might go wrong. Even if there's no evil person out there trying to misuse AI, bad things could still happen by accident, in a way that needs to be fixed by changing what AI gets built in the first place, not just testing it afterwards. If this was like making a car, we should install seatbelts and maybe institute a speed limit."

I think it's about salience. If you "feel the AGI," then you'll automatically remember that transformative AI is a thing that's probably going to happen, when relevant (e.g. when planning AI strategy, or when making 20-year plans for just about anything). If you don't feel the AGI, then even if you'll agree when reminded that transformative AI is a thing that's probably going to happen, you don't remember it by default, and you keep making plans (or publishing papers about the economic impacts of AI or whatever) that assume it won't.

I agree that in some theoretical infinite-retries game (that doesn't allow the AI to permanently convince the human of anything), scheming has a much longer half-life than "honest" misalignment. But I'd emphasize your paranthetical. If you use a misaligned AI to help write the motivational system for its successor, or if a misaligned AI gets to carry out high-impact plans by merely convincing humans they're a good idea, or if the world otherwise plays out such that some AI system rapidly accumulates real-world power and that AI is misaligned, or if it turns out you iterate slowly and AI moves faster than you expected, you don't get to iterate as much as you'd like.

I'm confused about how to parse this. One response is "great, maybe 'alignment' -- or specifically being a trustworthy assistant -- is a coherent direction in activation space." 

Another is "shoot, maybe misalignment is convergent, it only takes a little bit of work to knock models into the misaligned basin, and it's hard to get them back." Waluigi effect type thinking.

My guess is neither of these.

If 'aligned' (i.e. performing the way humans want on the sorts of coding, question-answering, and conversational tasks you'd expect of a modern chatbot) beha... (read more)

Oh, I see; asymptotically, BB(6) is just O(1), and immediately halting is also O(1). I was real confused because their abstract said "the same order of magnitude," which must mean complexity class in their jargon (I first read it as "within a factor of 10.")

That average case=worst case headline is so wild. Consider a simple lock and key algorithm:

if input = A, run BB(6). else, halt.

Where A is some random number (K(A)~A).

Sure seems like worst case >> average case here. Anyone know what's going on in their paper that disposes of such examples?

This doesn't sound like someone engaging with the question in the trolley-problem-esque way that the paper interprets all of the results: gpt-4o-mini shows no sign of appreciating that the anonymous Muslim won't get saved if it takes the $30, and indeed may be interpreting the question in such a way that this does not hold.

In other words, I think gpt-4o-mini thinks it's being asked about which of two pieces of news it would prefer to receive about events outside its control, rather than what it would do if it could make precisely one of the options occur,

Neat! I think the same strategy works for the spectre tile (the 'true' Einstein tile) as well, which is what's going on in this set.

Just to copy over a clarification from EA forum: dates haven't been set yet, likely to start in June.

Another naive thing to do is ask about the length of the program required to get from one program to another, in various ways.

Given an oracle for p1, what's the complexity of the output of p2?

What if you had an oracle for all the intermediate states of p1?

What if instead of measuring the complexity, you measured the runtime?

What if instead of asking for the complexity of the output of p2, you asked for the complexity of all the intermediate states?

All of these are interesting but bad at being metrics. I mean, I guess you could symmetrize them. But I feel like there's a deeper problem, which is that they by default ignore computational process, and have to have it tacked as extra.

I'm not too worried about human flourishing only being a metastable state. The universe can remain in a metastable state longer than it takes for the stars to burn out.

So at first I though this didn't include a step where the AI learns to care about things - it only learns to model things. But I think actually you're assuming that we can just directly use the model to pick actions that have predicted good outcomes - which are going to be selected as "good" according the the pre-specified P-properties. This is a flaw because it's leaving too much hard work for the specifiers to do - we want the environment to do way more work at selecting what's "good."

Second problem comes in two flavors - object level and meta level. The... (read more)

Multi-factor goals might mostly look like information learned in earlier steps getting expressed in a new way in later steps. E.g. an LLM that learns from a dataset that includes examples of humans prompting LLMs, and then is instructed to give prompts to versions of itself doing subtasks within an agent structure, may have emergent goal-like behavior from the interaction of these facts.

I think locating goals "within the CoT" often doesn't work, a ton of work is done implicitly, especially after RL on a model using CoT. What does that mean for attempts to teach metacognition that's good according to humans?

Would you agree that the Jeffrey-Bolker picture has stronger conditions? Rather than just needing the agent to tell you their preference ordering, they need to tell you a much more structured and theory-laden set of objects.

If you're interested in austerity it might be interesting to try to weaken the Jeffrey-Bolker requirements, or strengthen the Savage ones, to zoom in on what lets you get austerity.

Also, richness is possible in the Savage picture, you just have to stretch the definitions of "state," "action," and "consequence." In terms of the functiona... (read more)

I'm glad you shared this, but it seems way overhyped. Nothing wrong with fine tuning per se, but this doesn't address open problems in value learning (mostly of the sort "how do you build human trust in an AI system that has to make decisions on cases where humans themselves are inconsistent or disagree with each other?").

I honestly think your experiment made me more temporarily confused than an informal argument would have, but this was still pretty interesting by the end, so thanks.

I think there may be some things to re-examine about the role of self-experimentation in the rationalist community. Nootropics, behavioral interventions like impractical sleep schedules, maybe even meditation. It's very possible these reflect systematic mistakes by the rationalist community, that people should mostly warned away from.

It's tempting to think of the model after steps 1 and 2 as aligned but lacking capabilities, but that's not accurate. It's safe, but it's not conforming to a positive meaning of "alignment" that involves solving hard problems in ways that are good for humanity. Sure, it can mouth the correct words about being good, but those words aren't rigidly connected to the latent capabilities the model has. If you try to solve this by pouring tons of resources into steps 1 and 2, you probably end up with something that learns to exploit systematic human errors during step 2.

I give the probability that some authority figure would use an order-following AI to get torturous revenge on me (probably for being part of a group they dislike) is quite slim. Maybe one in a few thousand, with more extreme suffering being less likely by a few more orders of magnitude? The probablility that they have me killed for instrumental reasons, or otherwise waste the value of the future by my lights, is mich higher - ten percent-ish, depends on my distribution over who's giving the orders. But this isn't any worse to me than being killed by an AI that wants to replace me with molecular smiley faces.

One way of phrasing the AI alignment task is to get AIs to “love humanity” or to have human welfare as their primary objective (sometimes called “value alignment”). One could hope to encode these via simple principles like Asimov’s three laws or Stuart Russel’s three principles, with all other rules derived from these.

I certainly agree that Asimov's three laws are not a good foundation for morality! Nor are any other simple set of rules.

So if that's how you mean "value alignment," yes let's discount it. But let me sell you on a different idea you... (read more)

Yeah, that's true. I expect there to be a knowing/wanting split - AI might be able to make many predictions about how a candidate action will affect many slightly-conflicting notions of "alignment", or make other long-term predictions, but that doesn't mean it's using those predictions to pick actions. Many people want to build AI that picks actions based on short-term considerations related to the task assigned to it.

I think this framing probably undersells the diversity within each category, and the extent of human agency or mere noise that can jump you from one category to another.

Probably the biggest dimension of diversity is how much the AI is internally modeling the whole problem and acting based on that model, versus how much it's acting in feedback loops with humans. In the good category you describe it as acting more in feedback loops with humans, while in the bad category you describe it more as internally modeling the whole problem, but I think all quadrants ... (read more)

A process or machine prepares either |0> or |1> at random, each with 50% probability. Another machine prepares either |+> or |-> based on a coin flick, where |+> = (|0> + |1>)/root2, and  |+> = (|0> - |1>)/root2. In your ontology these are actually different machines that produce different states.

I wonder if this can be resolved by treating the randomness of the machines quantum mechanically, rather than having this semi-classical picture where you start with some randomness handed down from God. Suppose these machines us... (read more)

people who study very "fundamental" quantum phenomena increasingly use a picture with a thermal bath

Maybe talking about the construction of pointer states? That linked paper does it just as you might prefer, putting the Boltzmann distribution into a density matrix. But of course you could rephrase it as a probability distribution over states and the math goes through the same, you've just shifted the vibe from "the Boltzmann distribution is in the territory" to "the Boltzmann distribution is in the map."

Still, as soon as you introduce the notion of measure

Some combination of:

• Interpretability
  • Just check if the AI is planning to do bad stuff, by learning how to inspect its internal representations.
• Regularization
  • Evolution got humans who like Doritos more than health food, but evolution didn't have gradient descent. Use regularization during training to penalize hidden reasoning.
• Shard / developmental prediction
  • Model-free RL will predictably use simple heuristics for the reward signal. If we can predict and maybe control how this happens, this gives us at least a tamer version of inner misalignment.
• Self-modeling
  • M

The real chad move is to put "TL;DR: See above^" for every section.

When you say there's "no such thing as a state," or "we live in a density matrix," these are statements about ontology: what exists, what's real, etc.

Density matrices use the extra representational power they have over states to encode a probability distribution over states. If we regard the probabilistic nature of measurements as something to be explained, putting the probability distribution directly into the thing we live in is what I mean by "explain with ontology."

Epistemology is about how we know stuff. If we start with a world that does not inherent... (read more)

Treating the density matrix as fundamental is bad because you shouldn't explain with ontology that which you can explain with epistemology.

Be sad.

For topological debate that's about two agents picking settings for simulation/computation, where those settings have a partial order that lets you take the "strictest" combination, a big class of fatal flaw would be if you don't actually have the partial order you think you have within the practical range of the settings - i.e. if some settings you thought were more accurate/strict are actually systematically less accurate.

In the 1D plane example, this would be if some specific length scales (e.g. exact powers of 1000) cause simulation error, but as long ... (read more)

Fun post, even though I don't expect debate of either form to see much use (because resolving tough real world questions offers too many chances for the equivalent of the plane simulation to have fatal flaws).

With bioweapons evals at least the profit motive of AI companies is aligned with the common interest here; a big benefit of your work comes from when companies use it to improve their product. I'm not at all confused about why people would think this is useful safety work, even if I haven't personally hashed out the cost/benefit to any degree of confidence.

I'm mostly confused about ML / SWE / research benchmarks.

The mathematical structure in common is called a "measure."

I agree that there's something mysterious-feeling about probability in QM, though I mostly think that feeling is an illusion. There's a (among physicists) famous fact that the only way to put a 'measure' on a wavefunction that has nice properties (e.g. conservation over time) is to take the amplitude squared. So there's an argument: probability is a measure, and the only measure that makes sense is the amplitude-squared measure, therefore if probability is anything it's the amplitude squared. And i... (read more)