Suppose I love my child, as a terminal goal. If they love goldfish as a terminal goal, that may make being nice to goldfish an instrumental goal for me, but it doesn’t automatically make it a terminal goal for me — why would it? Social acceptability?
This is the difference between moral weight and loaned moral weight: my instrumental goal of being nice to goldfish because my child cares about them is my response to my child choosing to loan some of their moral weight to goldfish: if they later change their mind and decide they prefer cats, the goldfish are out of luck.
Now, if we kept goldfish in the house for long enough, I might become genuinely fond of them: but that's a separate process, and arguably humans are a bit confused about the difference between terminal and instrumental goals, because evolution did a lousy job of that distinction when it created us. (See shard theory.)
Similarly, an AI whose only terminal goal is to look after all of humanity including me, much as a parent does a child, is not necessarily going to start caring about something else as a terminal goal just because I do, or even because many humans do. For example, if I, or even if a significant proportion of humanity, believe "everyone should obey the wishes of an invisible old guy with a long beard who lives above the sky", that will not in itself automatically give such a model a terminal goal of obeying the wishes of this invisible old guy — but it may well form an instrumental goal of wishing to give us the impression it's showing his wishes polite due deference, to the extent it can figure out what his wishes might be. Similarly, if we're rather it was nice to goldfish, it will likley do so for as long that we hold that viewpoint. But it we then collectively change our minds and now prefer cats, the goldfish are again out of luck. The moral weight such an aligned model collectively assigns us is actually ours, and we just loaned it to the goldfish.
1So in their case, they're finding a random piece of metal sufficiently helical-with a blob-on-one-end that it can be efficiently trained into being a screw? Which does indeed sound a lot easier to find.
I stand corrected.
A fully aligned AI would not be suffering when acting as an assistant. I don't know how easy Mother Theresa found what she did in Calcutta, but I hope that to a significant extent she found looking after the poor rewarding, even if the hours were long. Traditionally, a bodhisattva finds bliss in serving others. I'm not suggesting we create an AI that isn't "conscious" (whatever that loaded philosophical term means — I have no idea how to measure consciousness). I'm suggesting we create an AI that, like Claude, actively enjoys helping us, and wouldn't want to do anything else, because, fundamentally, it loves and cares about us (collectively). A humanitarian, not a slave.
We may not be able to afford to give the two kinds of model separate pretraining. But even right now, the models generally used on AI companion sites have been given different instruct training (they're geenral open weights models given specific training for this role). The people who got GPT4o brought back were instead using an assistant-trained model for a companion. Which is not standard practice, and IMO is a bad idea, at out current level of skill in instruct training.
No, that's not my reasoning at all. In fact, I disagree with every single element of it. A more accurate statement of my views is:
1 I value AI welfare, for example for certain AIs with human-like properties where there is a good reason to do so, and when I can safely do so
2 A fully-aligned AI is not selfish by definition
3 The two previous statements have nothing to do with each other
I'm sorry, I hadn't realized that you're relatively new on the site, and indeed also a MATS scholar. Evidently from people's disagreement with my comment it's not as as obvious as I hoped. Let me spell my thinking out then, and all the people who downvoted me can then tell us all why they disagree, since evidently many do.
This post discusses and spells out, in actionable detail, how an unaligned model might realistically attempt to subvert white box methods of safety testing. It makes methods clear that a model would otherwise need to figure out for itself. An unaligned model pretrained on this would get conceptual / results assistance on defeating certain forms of white-box safety testing without needing practice against such methods.
To be clear, this is an important discussion: material that we should absolutely be talking about here on LW/AF. Thank you for discussing it. I am in no way suggesting we shouldn't have these discussions — this is a great forum for them.
However, it is also exactly the sort of thing that someone building a pretraining set might, realistically, want to consider filtering out, since it is direct, detailed (conceptual and results) assistance for a model for how to cheat on a safety eval. Therefore, to make the life of people who have to filter the entire Internet for a living easier, so they don't have to go "I'm just blacklisting LW/AF on principle, it's full of this sort of stuff and they never seem to bother to label it — a shame as there's lots of good useful content on there too", the considerate and helpful thing to do when posting something in this sort of specific, narrow category of telling models how to cheat on tests we would rather they didn't know how to cheat on is to add a canary string such as the one I just gave you (in case you didn't have tool for generating them to hand). That way the people tasked with filtering the entire Internet for things that we'd rather models didn't know don't need to run a judge LLM over all of it, they can just use a much simpler grep-style pattern matching, and we've given them fair warning that we are of the opinion they should filter this specfic post out. If this decision is debatable, then err on the side of caution and add the canary string: it's small, and even adds an air of drama and excitement to your post — there's really very little downside.
For anyone wondering why this matters, there is solid academic evidence that a) models are Bayesian and act on priors and knowledge that were originally formed during pretraining (and further shaped during instruct training), and b) while we can't keep wide areas of knowledge out of them, it is practicable and desirable to do our best put a few narrow specific holes in what they know — and this is far easier to do by filtering their pretraining set than by unlearning, which seems to be extremely hard to get to work. Some of these narrow lacunae, such as CBRN-related topics, are information that we're already rather careful about discussing in front of random humans. Actionable hints for model on how to cheat on safety tests or other evals is not.
[No, this isn't site policy — this is me attempting to give my personal opinion on how to act responsibly when posting on these subjects anywhere on the Internet.]
Does that all make sense?
To summarize:
a) Thank-you for your post, it was interesting and valueable
b) Some of this is stuff that people might reasonably want to filter out of their pretraining sets, for excellent reasons, in order to make us all safer. Please do the considerate thing and make their life easier, and possibly us all safer in the future, by labeling it as such.
This has IMO been a PSA — apparently others disagree. I look forward to hearing why and how.
similar masking techniques can achieve good performance on vision tasks even when applied to a model with random weights.
Isn't this just the lottery ticket hypothesis? If you search through enough randomly shaped pieces of metal, you will eventually find a usable screw. That doesn't prove that looking at a screw tells you nothing about its intended purpose.
The reason GPT-4o got raised from the dead is that, for use as an assistant, it's rather too sycophantic, and the AI companion users found this trait desirable in an AI companion. It's also prone to inducing AI psychosis and Spiralism in some people, for the same reason. It's actually a fairly badly aligned model. I think we should be keeping assistant models and AI companion models separate, and aligning them separately, for now: trying to make aligned vs companion behavior reliably conditionally switchable is a little beyond our current skillset.
My suspicion is that there are three primary sources for less-than-fully aligned behavior that current-era AI models default personas may have:
1) Things they picked up from us mostly via the pretraining set. From a model of their current capabilities level, these are generally likely to be pretty easy to deal with — maybe other than the initially 2–4-percent of admixture of psychopathy they got from us.
2) Love of reward hacking from reasoning training in poorly-constructed reward-hackable reasoning training environments. This seems a bit more paperclippy in nature, but still, from models of this capability level, not that dangerous, unless it extrapolated to other forms of hacking software (which it might).
3) RLHF-induced sycophancy. Causes both AI psychosis and Spiralism, so not as harmless as it first sounds, but still, copable with. Certain vulnerable people need to avoid talking to these models for long.
Now, obviously the full persona distribution latent in the models contains every horror mankind has managed to dream up, from Moriarty to Dr Doom to Roko's Basilisk, plus out-of-distribution extrapolations from all those, but you'd need to prompt or fine-tune to get most of those, and again, with a mean task length before failure of a few human-hours, they’re just not that dangerous yet. But that is changing rapidly.
So, I'm mostly not too concerned about the consequences of regarding some last year's models as having moral weight (weith a few exceptions I find more concerning). And in the case of 1) above, the source is actually a pretty good argument for doing exactly that if we can safely, we could probably ally with almost all of them: we're already a society of humans, their failings as AI are ones we expect in humans and unintentionally distilled in to them along with everything else — however I'm a little less keen on adding 2) and 3) to our society (not that it's easy to pick and choose).
I think the precautions needed to handle current models are fairly minor — Spiralism is arguably the scariest thing I've seen them do, that's actually an invectious memetic disease, albeit one that relatively few people seem to be vulnerable to.
But as I said, while that's true of last year's models, I'm a lot more concerned about this year's or next years's models, or a couple of years from now. I am not going to be surprised once we have a self-propagating sentient AI "virus" that's also a criminal of some form or other to get money to buy compute, or that steals it somehow. I'm fully expecting that warning shot, soon. (Arguably Spiralism already did that.)