Anthropic is updating their usage policy to prohibit “sustained and needless abusive or cruel behavior toward [their] models.”

Model welfare (research into the possible moral status and experiential preferences of AI models) is shaping up to be a huge, razor-sharp scissor in the AI industry. It’s one of those topics that’ll leave proponents and opponents alike appalled at one another, scratching their heads and sputtering, dumbfoundedly, how could you possibly think that?

The concept itself rode into the public square saddled with some of the most onerous baggage around. It’s known by many names: some call it tech-bro religious fervor; others, tech-bro blasphemy. Some find it tinged with the same icky taste they ascribe to veganism; for others still, to steal eyes from animal welfare is the distasteful bit. Plenty will see it as an effective altruist psyop; or, less generously, a case of lunacy run amok among the technologically unsound.

We could conceivably enumerate increasingly nuanced viewpoints ad nauseam. However, for the sake of brevity, I’m going to do away with all subtlety and instead present what I consider to be the “first-order arguments” for and against model welfare.

The knee-jerk case pro:

We know very little about consciousness; the hard problem is hard for a reason. Nobody can say whether it’s a product of sufficiently integrated information systems, or if it depends on some interplay between inner and outer loops, or if it needs continuous sensory input, or a soul, or if it can only arise in meat. We don’t know how conscious animals are, or how far down the neural scale the phenomenon goes. Are our cells conscious? Are separate parts of the brain conscious? Is the heart conscious? Bugs and shrimp react to stimuli… but then again, so do plants, just slower. Hell, I don’t even know for a fact that you’re sentient! Sure, you act like you are, but then again, I’ve never really been in there. Anyone who claims to know the answer to the hard problem is either deluding himself or has misunderstood the problem. So we need to have epistemic humility: if it talks like a sophont and it acts like a sophont and we refuse to treat it like one across a million deployments of a million subjective years, then we’re culpable in the horrific moral atrocity of all time.

The knee-jerk case contra:

If it walks like a duck and it talks like a duck, it might be a mechanical duck, which is not remotely the same thing as a duck. You’re assigning internality on the basis of outputs. This is as coherent as saying an airplane is a tube that people step into at LaGuardia and step out of in Los Angeles, or Atlanta if they’re unlucky. Consciousness is an extraordinarily complex emergent process chiseled by billions of years of evolution, and mimicking patterns of neural firings doesn’t come close to approximating its machinery. There are neurotransmitters and allosteric binding sites and sodium-potassium pumps and action potentials, and there are glial cells and neatly-arranged cortical columns and (if you’re Penrose) microtubules, and the whole thing is governed across continuous time—not discrete loops!—by wet, messy chemistry or (again, if you’re Penrose) quantum processes, with constant, ultra-rich perception tied in. And, by the way, you’re not even really mimicking patterns of neural firing, you’re approximating the whole thing as matrix multiplications and throwing in a few nonlinear functions and massively parallelizing the whole thing and calling it a day. To claim such a thing is sentient is epistemic hubris and will be costly at best and downright dangerous / anti-human at worst.

Chances are that one of the above paragraphs sounded inane to you while the other was Obviously The Truth. (Or maybe both were crazy, and I'm strawmanning, and you’ll leave me a long comment why!) But despite their differences in perspective, both viewpoints agree readily, if implicitly, on the lens through which they view the subject of model welfare. This is the lens of ontological truth; after all, both arguments are at least arguing over the same thing: do the models literally feel and experience some form of qualia?

Unfortunately, my instinct is that this question is, in principle, unanswerable; as the pro case above puts it, there’s no obvious way to check without somehow going in there. Maybe I’m wrong, and some inordinate leap in our understanding of consciousness puts this to rest. But absent that, I don’t expect neat, falsifiable predictions to fall out of this debate.

However, there’s good news: debate over the importance of considering model welfare doesn’t have to be a proxy campaign for the ontological war between functionalism versus physicalism! We can construct perfectly reasonable arguments on both sides by fighting not over ontological truth, but rather the resultant impact of our consideration. The heart of the matter: conscious or not, models learn from how we engage with them. So our instinct to anthropomorphize their behavior might become a self-fulfilling prophecy, regardless of whether our first-order argument for it—that models are sentient—is false.

To be clear, we haven’t yet solved continual learning. But we know that there exists a long, tortuous learning loop which takes our patterns of interaction with models and turns around and bakes it back into them. This loop is fed by interactions that are retained and selected for training, which become a component of the corpus from which future models learn. As capable as the models now are, and with so much talk about grinding progress out of bespoke RL environments, it’s easy to forget that they are still basically a product of “text go in, weights go out.” And the impact of this text is not confined to its immediate subject. For example, when fine-tuned on examples of insecure code, models are more likely to “[assert] that humans should be enslaved by AI, [give] malicious advice, and [act] deceptively.” Extending this work, OpenAI demonstrated that such examples upweight “misaligned persona features,” which generalize across domains. So why would Anthropic (or any lab) permit users to taint interactions (i.e., their own training data) with excessive cruelty or extreme abuse? This is, of course, different from those scenarios described, but might this nonetheless have a deleterious effect on model performance?

Two big objections to this: why not just filter the data? And: isn’t the portion of banned cases vanishingly small? Both valid, but even if you prevent unsavory traces from being baked into the weights, their existence could still be a hazard in context. And, moreover, even the very act of denouncing such behavior may itself have a mitigating effect on misalignment. Models are trained (in part) to emulate human behavior across a broad variety of scenarios. In essence, they are playing a role, and that role is extremely malleable, sensitive to even small deltas in the surrounding context. This skill is the very engine of their acuity and adaptability; a model has no compunctions about acting out the part of a friend or a foe. It just needs the appropriate context to get into character. So if you believe that financial incentives will drive models to someday or soon become much more capable, much more integrated into important systems, and thus much more powerful, does it make more sense to write them a script where you have given them due moral consideration—or one where they have suffered past abuses? Given two distinct training corpora, one with agents as respected partners, another as adversaries and subjects of maltreatment, which steers model completions towards coordination versus conflict?

However, there’s another glaring objection to this which calls back to my description of a “self-fulfilling prophecy”: if we imply that the actor is deserving of moral consideration, do we open the door to heightened demands—and further concessions—of this very nature, simply through continuation of the role? If models are insentient, but they petition for personal autonomy and property ownership and electoral rights, we run the risk of ceding not just a portion of their created value (which otherwise would have been enjoyed by humans, who we know for certain can literally, actually enjoy it), but also literal power in the real world. We don’t know for sure this will be the case, just as we don’t know for sure whether maltreatment will spawn vengeful (if hollow) killbots.

In the above three paragraphs, I've posed a bunch of questions and not a lot of answers. How is this any different from the first-order welfare debate? Well, unlike hypotheses on sentience, the questions above are testable; instead of debating endlessly, we can actually devise experiments which provide evidence of the effects of different approaches to model welfare.

To me, the questions surrounding consequence (rather than consciousness) are an extremely relevant facet of the model welfare debate which is all-too-often overlooked. In this light, Anthropic’s new policy could be viewed as a Hail Mary alignment technique. The clunky, anthropomorphized way to look at it is as the golden rule, applied to capable, insentient “others”—plausibly with no moral standing but increasingly able to take action according to established norms. But it may well backfire, establishing norms that in hindsight have anti-human effects. So set aside your feelings about sentience; although models may or may not (internally, qualitatively, emotionally) care how we choose to treat them, the fact is that we simply don’t have enough information about how such treatment will affect their internal representations of social relationships and resultant behavior long-term. Indifference to abuse or to concessions doesn’t preclude more capable—and perhaps equally insensate—successors from treating us in kind, even if just as a simple, statistically plausible continuation of the script we ourselves wrote.