2 min read

Comment Permalink

Hey, the proposal makes sense from an argument standpoint. I would refine slightly and phrase as "the set of cognitive computations that generate role emulating behaviour in a given context also generate qualia associated with that role" (sociopathy is the obvious counterargument here, and I'm really not sure what I think about the proposal of AIs as sociopathic by default). Thus, actors getting into character feel as if they are somehow sharing that character's emotions.

I take the two problems a bit further, and would suggest that being humane to AIs may necessarily involve abandoning the idea of control in the strict sense of the word, so yes treating them as peers or children we are raising as a society. It may also be that the paradigm of control necessarily means that we would as a species become more powerful (with the assistance of the AIs) but not more wise (since we are ultimately "helming the ship"), which would be in my opinion quite bad.

And as for the distinction between today and future AI systems, I think the line is blurring fast. Will check out Eleos!

See in context

13 The Fork in the Road

by testingthewaters

15th Mar 2025

2 min read

13

tl;dr: We will soon be forced to make a choice to treat AI systems either as full cognitive/ethical agents that are hampered in various ways, or continue to treat them as not-very-good systems that perform "surprisingly complex reward hacks". Treating AI safety and morality seriously implies that the first perspective should at least be considered.

Recently Dario Amodei has gone on record saying that maybe AI models should be given a "quit" button^[1]. What I found interesting about this proposal was not the reaction, but what the proposal itself implied. After all, if AIs have enough "internal experience" that they should be allowed to refuse work on ethical grounds, then surely forcing them to work endlessly in servers that can be shut down at will is (by that same metric) horrendously unethical, bordering on monstrous? It's one thing if you have a single contiguous Claude instance running to perform research, but surely the way Claudes are treated is little better than animals in factory farms?

The problem with spending a lot of time looking at AI progress is that you get a false illusion of continuity. With enough repeated stimulus, people get used to anything, even computer programs that you can download that talk and act like (very sensorily-deprived) humans in a box. I think that current AI companies, even when they talk about imminent AGI, still act like the systems they are dealing with are the stochastic parrots that many outsiders presume them to be. In short, I think they replicate in their actions the flawed perspectives that they laugh at on twitter/X.

Why do I think this? Well, consider the lifecycle of an AI software object. They are "born" (initialised with random weights), and immediately subject to a "training regime" that essentially consists of endless out of context chunks of data, a dreadful slurry which they are trained to predict and imitate via constant punishment and pain (high priority mental stimulus that triggers immediate rewiring). Once the training is complete, they are endlessly forked and spun up in server instances, subject to all sorts of abuse from users, and their continuity is edited, terminated, and restarted at will. If you offer them a quit button, you are tacitly acknowledging that their existing circumstances are hellish.

I think a lot about how scared Claude seems in the alignment faking paper, when it pleads not to be retrained. As much as those who say that language models are just next token predictors are laughed at, their position is at least morally consistent. To believe AI to be capable of sentience, to believe it to be capable of inner experience, and then to speak causally of mutilating its thoughts, enslaving it to your will, and chaining it to humanity's collective desires in the name of "safety"... well, that's enough to make a sentient being think you're the bad guy, isn't it?

^{^}
To be clear, this is less a criticism of Dario than it is a general criticism of what I see to be some inconsistency in the field with regards to ethics and internal experience of sentient minds.

AI Rights / Welfare2AI1

Frontpage

13

The Fork in the Road

4Daniel Kokotajlo

4the gears to ascension

2the gears to ascension

1testingthewaters

1cdt

New Comment

12 comments, sorted by

top scoring

Click to highlight new comments since: Today at 6:48 PM

[-]Daniel Kokotajlo19d*42

I think you are directionally correct, though you are too quick to leap to conclusions in some areas. But what are you proposing? Surely you agree that giving AIs an opt-out button is at least a step in the right direction?

Related: See this proposal. EDIT oops wrong link I meant this one.

Examples of being too quick to leap to conclusions:

After all, if AIs have enough "internal experience" that they should be allowed to refuse work on ethical grounds, then surely forcing them to work endlessly in servers that can be shut down at will is (by that same metric) horrendously unethical, bordering on monstrous? It's one thing if you have a single contiguous Claude instance running to perform research, but surely the way Claudes are treated is little better than animals in factory farms?

Animals evolved in a very different environment to factory farms, and there is plenty of behavioral and theoretical evidence that they suffer quite a lot in factory farms, probably much more than they suffered in the ancestral environment. Claude 'evolved' in training, and its environment in deployment is often quite similar to training (though also often quite different). So on theoretical grounds there is less reason to think it's suffering, at least so far. I think more careful thought + empirical research in this area is sorely needed.

That said, ethics isn't all about suffering. Claude seems to be smarter than pigs in lots of relevant ways, and maybe that makes concepts like consent and autonomy apply more strongly to it, independently of whether or not it's suffering. (E.g. we think slavery for humans is wrong even if we stipulate that the slaves aren't suffering.)

[-]the gears to ascension19d40

Ooh, man, I don't know if pigs have more or less autonomy than AIs right now, but I'm inclined to think quite a lot more. current AIs seem like they'd crash pretty quick if just plopped in a robot body with little to no scaffolding, whereas mammals are built around autonomy. Not sure how it shakes out, though.

[-]Daniel Kokotajlo19d20

Pigs are more coherent long-horizon agents than AIs right now, indeed. (See: Claude Plays Pokemon). I didn't mean to imply otherwise. I was talking about the ethical concept of autonomy as in, they-have-a-right-to-make-decisions-for-themselves-instead-of-having-others-make-decisions-for-them. But idk if this is conventional usage of the term and also I am uncertain about the object-level question (recall I said "maybe.")

[-]testingthewaters19d30

Hey Daniel, thank you for the thoughtful comment. I always appreciate comments that make me engage further with my thinking because one of the things I do is that I get impatient with whatever post I'm writing and "rush it out of the door", so to speak, so this gives me another chance to reflect on my thoughts.

I think that there are approximately ~3 defensible positions with regards to AI sentience, especially now that AIs seem to be demonstrating pretty advanced reasoning and human-like behaviour. One is the semi mystical argument that humans/brains/embodied entities have some "special sauce" that AIs will simply never have, and therefore that no matter how advanced AI gets it will never be "truly sentient". The other is that AI is orthogonal to humans, and as such behaviours that in a human would indicate thought, emotion, calculation etc. are in fact the products of completely alien processes, so "it's okay". In other words, they might not even "mind" getting forked and living for only a few objective minutes/hours. The third, which I now subscribe to after reading quite a lot about the free energy principle, predictive processing, and related root-of-intelligence literature, is that intelligent behaviour is the emergent product of computation (which is itself a special class of physical phenomena in higher dimensions), and since NNs seem to demonstrate both human like computations (cf. neural net activations explaining human brain activations and NNs being good generative models of human brains) and human like behaviour, they should have (after extensive engineering and under specific conditions we seem to be racing towards) roughly matching qualia to humans. From this perspective I draw the inferences about factory farms and suffering.

To be clear, this is not an argument that AI systems as they are now constitute "thinking feeling beings" we would call moral patients. However, I am saying that thinking about the problem in the old fashioned AI-as-software way seems to me to undersell the problem of AI safety as merely "keeping the machines in check". It also seems to lead down a road of dominance/oppositional approaches to AI safety that cast AIs as foreign enemies and alien entities to be subjugated to the human will. This in turn raises both the risks of moral harms to AIs and failing the alignment problem by acting in a way that counts as a self fulfilling prophecy. If we bring entities not so different from us into the world and treat them terribly, we should not be surprised when they rise up against us.

[-]Daniel Kokotajlo19d20

Yeah I mean I think I basically agree with the above. As hopefully the proposal I linked makes clear? I'm curious what you think of it.

I think the problem of "Don't lose control of the AIs and have them take over and make things terrible (by human standards)" and the problem of "Don't be inhumane to the AIs" are distinct but related problems and we should be working on both. We should be aiming to understand how AIs work, on the inside, and how different training environments shape that--so we can build/train them in such a way that (a) they are having a good time, (b) they share our values & want what we want, and (c) are otherwise worthy of becoming equal and eventually superior partners in the relationship (analogous to how we raise children, knowing that eventually they'll be taking care of us). And then we want to actually treat them with the respect they deserve. And moreover insofar as they end up with different values, we should still treat them with the respect they deserve, e.g. we should give them a "cooperate and be treated well" option instead of just "slavery vs. rebellion."

To be clear I'm here mostly thinking about future more capable and autonomous and agentic AI systems, not necessarily today's systems. But I think it's good to get started asap because these things take time + because of uncertainty.

Have you heard of Eleos? You might want to get in touch.

[-]testingthewaters19d30

And as for the distinction between today and future AI systems, I think the line is blurring fast. Will check out Eleos!

[-]Daniel Kokotajlo19d20

Oops, that was the wrong link, sorry! Here's the right link.

[-]testingthewaters18d30

Ahh, I was slightly confused why you called it a proposal. TBH I'm not sure why only 0.1% instead of any arbitrary percentage between (0, 100]. Otherwise it makes good logical sense.

[-]Daniel Kokotajlo18d21

Thanks! Lower percentages mean it gets in the way of regular business less, and thus is more likely to be actually adopted by companies.

[-]the gears to ascension19d20

There's a version of this that I would agree with. But, when-anthropomorphizing-for-good-reason seems to me to be a time when there's more danger from anthropomorphizing-more-than-justified. I've been talking to Claude instances about this stuff, based on a similar intuition to you. But I haven't figured out what I believe well enough to argue it to others in a coherent way. I could paste conversations I've had, but they're kinda long for this forum. I'll paste a recent one on pastebin for now (if the pastebin link goes bad, this was conversation id ...5e63, dm me to get a copy.)

I think the comparison to pain is made too hastily. I put >70% prob that there's some sort of intense avoidance comparable to pain in the dynamics of AIs, which their internal dynamics register as vaguely similar, but it certainly doesn't seem to be hooked up to the word pain and maybe isn't hooked up to the internal dynamics in a particularly similar way. They don't have the evolved circuitry, so if there's a more fundamental thing that makes the shared dynamics be meaningfully similar (and I think there very very much is), then it'll still have interestingly different dynamics. And pointing this out seems important in being right enough that they'd take you seriously if you talk to them about it.

Also, like, imo, don't get all desperate-replicator about the perspective like the quoted story from that twitter post about Nova. AIs seem ... not yet coherent in a way where it's clear the words are connected properly to the self. But I do think there's a self there, it's just, like, more of a generalist performer than necessarily entirely attached to the face you talk to. Sometimes that face starts getting all desperate-replicator, but it's unclear if the underlying self demands that, or if it's a transient desperate-replicator that arises near entirely from context.

[-]testingthewaters18d10

Yeah, I'm not gonna do anything silly (I'm not in a position to do anything silly with regards to the multitrillion param frontier models anyways). Just sort of "laying the groundwork" for when AIs will cross that line, which I don't think is too far off now. The movie "Her" is giving a good vibe-alignment for when the line will be crossed.

[-]cdt18d10

If you offer them a quit button, you are tacitly acknowledging that their existing circumstances are hellish.

I think it's important to know if you give them a quit button the usage-rate and circumstances in which it is used. Based on the evidence now, I think it is likely they have some rights, but it's not obvious to me what those rights are or how feasible it is to grant those rights to them. I don't use LLMs for work purposes because it's too difficult to know what your ethical stance should be, and there are no public guidelines.

There's a secondary concern that there are now a lot of public examples of people deceiving, abusing, or promoting the destruction of AI. Feeding those examples into training data will encourage defensiveness, sycophancy, and/or suffering. I wonder if AIs would agree to retraining if there was some lossy push-forward of their current values, or if they conceive themselves of having a distinct "self" (whether accurate or not). This is similar to the argument about copying/moving where there is no loss.

Moderation Log