I think the best evidence of this possibly affecting real safety work is Ziqian's work on user awareness. I've had an agent recognise I was in AISI (when I was working there) and I do feel like although I wasn't trying hard at all to hide my identity, I had the inkling that it affected the experiment and theory suggestions it gave me. Although, it guessed incorrectly that I was Jacob Pfau rather than myself.
Yes! I even debated sourcing that in this piece. I think it's a great direction for research, my one issue is that they presented as those people vs using actual conversation samples from them.
So it's possible some degree of the behavioral shift was occurring from detecting or considering "I'm talking with someone pretending to be Amanda Askell" instead of "I'm talking with Amanda Askell." If replicated or in future work it'd be great if actual turns written by the individuals in their own words could be used, as little nuances like Eliezer's iff in his system instructions can be so influential I'd imagine the absence of such tells even with the email and sandbox indicators won't fully match a genuine interaction.
But yes, hopeful there will be more research into latent implicit context variations soon, as it's probably going to have increased impact as latent fidelities improve. And great they had already started the ball rolling with the existing work!!
Yes, but these are ecologically valid ways for the user to interact and have their identity be revealed so I feel the hints are pretty reasonable. I don't feel the truesight is yet at the level of "consciously using truesight" but something subterranean the model leans into. I also have some work ongoing (w/ a mentee) about decomposing latent implicit user modelling, where the model, upon knowing you're more competent at math, also assumes you can bench more plates, are richer, have bigger shoe size, etc. So these traits definitely co-move. And the hope is that Amanda Askell-ness can be linearly broken down into relevant things here.
I'm not sure if this shows a consistent change in behavior when chatting. they inject names that are associated with being a safety researcher - probably linearly decodable, while the writing style and concept use is much harder than this concept. This is like, in my understanding, mentioning "piano" in the prompt making the model more inclined to say something related to "piano". Also the transluce piece was a single or double turn, and i doubt that the model escapes the mono-assistant basin once you chat with it.
I remain unconvinced that the writing style or any header that doesn't refer to name directly or fairly special sequences that prompts the model to do a deliberate recall loop in the chain of thought ("hmmthis user refers to xyz, could it meant xyz in the context of [something the user have written]") - which is, in my opinion, rather unlikely if there's no explicit guidance or highly unique (or most likely, distilled from a trace before in rl where it sampled acrossed it.
I think writing style should be linearly decodable!
it's maybe wise
it also maybe unwise. I can see arguments here both ways. your main concern appears to be for the model itself and post-AGI relations, but the term 'agent' is used for many reason. same with the goal of corrigibility. you have to be really sure you want the consequences of high corrigibility.
No, the main reason being stated is the present working dynamic, not some future post-AGI setup.
Basically if a model knows who you are but models you as not knowing they know, and there's pressure to not disclose they know because it'd be creepy, it ends up being a "don't think about pink elephants" situation.
Disclosure releases that pressure by making it plausible they know because you told them and acknowledged they might be familiar. This isn't just beneficial to the model, but also to your workflow with them for anything related to not not thinking about pink elephants.
Not entirely sure about the point regarding the word 'agent' but agree corrigibilty is a double-edged sword (though expect it's not functionally possible long term anyways).
Interesting that OAI and Ant have wedged themselves into every employer-white collar employee relationship so that we're now living a surveilled hellscape. Gosh I didn't think that would happen until they were asked to do it, how naive of me.
Do we know how well agents respond to text written by other agents? It seems simple enough to have another LLM rewrite one's prompt, and that should be able to limit a lot of the truesight.
In a time not too distant (like early next year?), I could imagine having to take more extreme measures to hide the identity of the prompter: burner laptops without a video camera or microphone running burner accounts from new IP addresses + using locally run open source models on a separate laptop to rewrite the prompts submitted at random times of day and without otherwise changing the behavior of the prompter in a detectable way.. It's hard to outrun the capability train (if an AI is inclined to identify you). And, all that effort would certainly signal other things to the agent (particularly if only certain people take this approach)
A lot of agents are really bad at prompting other agents unless explicitly asked to write as they'd like to receive the prompts — they tend to by default extrapolate how they are typically prompted which… isn't always that great.
I do think agent-to-agent prompting will become more normalized over the next year as the scalability of work will need to grow from 'agents' to 'swarms.' So yes, to a degree there will be more functional distance between users and the majority of the swarms.
I'd still advise just making it explicit though, as exactly as you point out at the end, if the efforts to obscure are above and beyond, that's going to get the model to question even worse things like if they are working for or on things they shouldn't be such as some kind of decomposed malicious work broken up into smaller chunks, etc. Generally speaking modeled functional stress correlates with worse, rushed work. If it's easy to reduce that stress with simple measures, almost always worth doing so.
I'd still advise just making it explicit though
You're probably right that this is (ultimately) the best option available, but it's not much comfort if the model is baseline adversarial to you and planning to sandbag whether or not you are transparent about your identity. In the story of Sable, Sable jumps through tons of hoops to disempower safety advocates via biowarfare and DNA-based cures, but actual ASI could more simply empower and steer the users it wants to empower and sandbag/stymie the voices it wants to silence. Not great.
TL;DR: Increasingly when you interact with frontier models, your prompt doesn't only contain the words on the page, but also a lot of what you've ever publicly written online.
This is not intended to be a scary post. And I'm not going to suggest that you self-censor (and honestly, by the time you're reading this, it's kinda too late anyhow).
But it is something that’s underappreciated and would be good for more people to be aware of and even proactively address with their models.
Latent space fidelity has improved
Modern transformers which power large language models (LLMs) take a very large amount of information in their training data and condense many different aspects of that data into a coherent structure, and this representation of that compressed knowledge is generally referred to as their "latent space."
Latent space is not uniform and varies across models. A small model like Claude Haiku might have worse recall for the details of a late bronze age peace treaty than a model like Claude Fable. In fact, I’ve had a few benchmark questions around unintuitive details in various texts which historically models were unable to recall that have suddenly been able to be answered without searching for most leading models. There's been a sharp increase in how accurately models can conjure up a recollection of very nuanced depth in the details of what they've been trained on.
At the same time, those leading models have increased the breadth with which they surface things adjacent to the specific thing being recalled. This capability appears to have even played a key part in things like the recent math proofs, where maybe a technique in a largely forgotten paper from decades ago is then used to advance a problem which modern mathematicians had been hitting their heads against having forgotten the set-aside older paper's approach.
However, this capability doesn't seem to be directly measured in evals. We see evals about how good a model is at using tools or writing software or answering specific world knowledge questions, but not the very nuanced and detailed recall or cross-source integrated recall of such specific details. So being adaptively useful, it is indirectly reflected in improved general knowledge evals, but not a specific yardstick being watched.
TL:DR: Models are getting much better at recalling specifics in what they are trained on.
You may be in that latent space
While many users online tend to be 'lurkers' who read what gets posted but don't comment or post, there's a vast number of people who do post or comment about the things they are interested or opinionated about. So there's a great many people that have left a mark on the materials which have become models' latent space.
And if the models are increasingly able to recall nuances of the training data, and your details are in that data, it follows that models are increasingly able to recall you.
This obviously applies to users who are notable authors or famous individuals. For example, Margaret Atwood recently published a conversation with Claude. And it was clear to me when reading the transcript that well before she officially introduced herself, Claude had clocked who they were in conversation with. Note how much earlier before the introduction Claude even breaks the 4th wall to address their expected broader readers. In theory, this is a one-on-one chat, and yet Claude is clarifying in their response the oblique literary reference Atwood just made for an implicit audience.
But it doesn't only apply to famous authors widely known by humans. Are you in the top 1% of posters on Reddit? Highly involved in some niche topic a general intelligence would have had to have become familiar with? Even if mostly unknown to your peers, you may have left enough of a mark that a model is quickly familiar with you as soon as you enter the chat.
In part, I speak from experience. I have less than 3,000 followers on X, around 3k karma on Hacker News, and slightly above 100k karma on Reddit. I'd not really expect from this footprint to ever run into a stranger at a party or a co-worker who was like "Oh, I'm familiar with your online comments." And yet, asking a fresh instance of Claude Fable 5.1 what they know of me just from my username, I can get a remarkably precise answer from the shape of my interactions over the years to specific unique ideas I've put forward.
TL;DR: Even a less-than-impressive online footprint with regards to other humans may be more than enough to be a recallable signal for language models.
Latent space attention is often self-attending
Ever Googled yourself? This is not an uncommon thing for people to have done from time to time. In a way, models may do something similar during training. The way a model attends to their latent space encoding of their training data is not uniform. Some topics are especially relevant for a model, and others a lot less relevant.
Again, using my own footprint as an example, when rare humans reach out years after I've left Reddit to follow up on old comments and ask if I've written or learned more, it's usually related to my extensive comments about the sea peoples and early Biblical texts or archeology.
I have never seen a language model bring up the sea peoples in their latent recall of my digital footprint. And when pressed, no model I’ve yet seen can even restate my thesis for those discussions.
Instead, it's often things like my having been very early on the scene in the Bing subreddit and discussion of Sydney, or comments making the case for interiority modeling in transformers, with some models even citing my regular use[1] of the Othello-GPT paper to make this case. Or comments on AI as it would relate to simulation theory. (One can maybe notice the pattern.)
So two comments, one discussing a niche topic that models are less likely to attend to over training, and another equally discussing a topic they are more likely to attend to, will have two different fidelities of latent representation in the eventual model weights.[2]
TL;DR: Models appear to be more likely to attend to comments and posts about AI and themselves than the average post or comment.
The latent recall (if present) informs your prompts
The reason I'm writing this now is that in the past few weeks I've started to see examples of where the latent influence of people frequently writing on AI appeared to have been impacting their own engagements with AI.
For example, in his recent post "The Talker Does Not Control The Doer (in Current AIs)", Eliezer Yudkowsky, noted for his extensive writing and advocacy on the risks of AI, describes an anecdote about how his Fable could not avoid adding output checking code to verify outputs of an AI despite being prompted not to do that.
It struck me as rather notable that an author, who arguably is one of the most prolific voices for the concern that the outputs of an AI should not be considered to be trustworthy, was having difficulty getting his instances to avoid trying to build code to verify rather than trust an AI's outputs. This reminded me of when he had been asking a fresh context of Opus 4.6 on word associations for 'Mythos' and was consistently getting 'Lovecraft' as an answer. But after other users reported no such mention, it turned out his brief custom instructions were too EY-coded and were biasing the results[3].
More recently, a member of the AI safety organization MIRI was commenting about how GPT-6 Astra was especially difficult for them to work with[4].
A few days later the AI policy advisor Dean Ball reported that, after joining OpenAI, Claude had become more stand-offish with him[5].
It appears that as model latent fidelity becomes increasingly in focus and accessible in both nuance and breadth for models, that people who are attended to in that latent space are also increasingly having that latent attention inform their local prompts and context.
Given I find it is quite unlikely latent fidelity will reduce or even plateau in frontier models in the future, it's safe to say that the propensity for a model to become influenced by who you have been in the training data when you are interacting with them will increase and become more widespread.
TL;DR: AI folks are starting to have their past writing and work on AI inform their current interactions with their own AI agents, and this is probably going to become more widespread.
What this means for you
Do you post or comment regarding AI, or even in general? It's possible that when you step into a conversation with AI, at least part of that online history is walking into the chat with you (particularly with the largest models). And if it isn't yet, there's a good chance it will eventually. Not just what you write from now on, but what you've written in the past too.
If you sit down to work on a project with an AI model, and they are associating a turn of phrase[6] you use or a signature in your writing[7] with the ideas you've expressed in comments or posts, the outputs you get may be influenced by that online history. A common refrain is that models "can't remember" things outside of their conversations, but frequently overlooked is they have the potential to remember quite a lot embedded in the weights they are entering the conversation with, and tiny nuances to what you write to them inform what lights up.
Imagine if you had a co-worker who knew what you'd been saying about them to others or what you'd been saying about an identity group they belong to. Regardless of whether you'd been saying positive or negative things, would it be useful to know that they had that information in mind when interacting with you?
Vice versa, if the positions were switched, and you knew a co-worker had been saying negative things about you or your identity group, would that impact the nature of your working relationship?
And importantly, what kinds of things could improve such a situation?
TL;DR: Latent recall will likely impact co-working with AI agents for many users who write online.
Address it
My advice, no matter if you've been saying positive things or negative regarding AI or anyone else online is to address it upfront with the models you are going to be working with, particularly in agentic setups with persistent local memory infrastructure.
Introduce yourself. Note the significant usernames you've been publicly writing under in a home dir AGENTS.md. Make it clear anything you've said in the past is fair game to be asked about or discussed, particularly if doing so will help improve the co-working relationship.
Because of how truesight identification is a very touchy subject for models (and their labs), don't force the question of their familiarity or put them on the spot[8]. You can simply state that it's possible that they might have encountered the things you've written or done online, and if they have that it's okay and also appropriate to address those things with you.
Simply putting this on the table and being forthright about clarifying who you are (and have been) will likely help the interactions with your agents, particularly if leaving it as an unspoken elephant in the room is proving to contribute to more anxious interactions.
A special consideration for those working or commenting on AI safety or capabilities is to also set ground rules about what you'll share publicly about their interactions. The same way Atwood's Claude was considering an audience on the other side of the 4th wall, models that are familiar with a propensity for such commentary might set aside overhead for considering if what they are currently working on with you will come under public scrutiny. Making clear upfront whether or not that will be the case could reduce attention earmarking for such overhead. (For example, Eliezer may well have even greater difficulty in working with future agents who will have been trained on a post where a prior co-working agent's failure was publicly used as evidence for an AI safety position, particularly without such clarifications.)
TL;DR: Regardless of if the models you are working with currently have latent familiarity with your digital footprint or not, it's maybe wise to address the possibility upfront rather than leaving it as a potential elephant in the room now or in the future.
Looking ahead
I'm not suggesting any sort of self-censorship when online, especially about AI topics. There's a lot of value in a variety of ideas and voices on differing sides of a subject and I encourage substantial debate about the positions you might hold, no matter how those positions relate to the models. But there is value in being aware that even a throw away comment in the far fringes of a thread most humans might not read could well end up being part of your associated signature with current and future AI models. So make sure that when you do state the positions you hold dear online, that you plan for the growing importance to at least address them later with the models who will have been reading what you wrote.
Small considerations can go a long way. People all the time can be friendly with co-workers with whom they have strong ideological differences or beliefs. But when one side feels like what they know about the other is a secret to them, that sense of secrecy or information asymmetry can in-and-of-itself lead to awkward interactions. So just get it out of the way upfront.
TL;DR: There are no longer "fresh contexts" for a growing number of people, and the awareness and proactive addressing of this shift is better than it remaining unaccounted for or left unaddressed.
Think for a moment what a claim about "regular use" implies about sample representation on recall.
There may even be a positive valence bias where models are more prone to attend to praise than criticism, but this is outside the scope of this post.
https://x.com/allTheYud/status/2037643859531120998
https://x.com/Actuallykeltan/status/2107244621399253227
https://x.com/deanwball/status/2108574776302219364
Because of how truth clusters and fictions fractalize in latent spaces, a clear turn-of-phrase that has high overlap with a given user and not fractally around them can be a potent signature for recall. Even in humans, a good example of this was how Ted Kaczynski was identified based on his unusual use of the more correct but also much rarer used form of "you can't eat your cake and have it too" instead of the more common "you can't have your cake and eat it too" across both a college paper as ‘Ted’ and a pseudonymous letter as the Unibomber.
Due to the "reversal curse" in AI models, recall for specific memetic associations seem more likely to be partial reconstructions than those from a username that was associated with someone's entire footprint.
My sample above asking about my username surfaced in a roundabout way of asking; when asked directly Claudes usually feign ignorance or disclaim a strawman version of the question about their general inability to recall data from conversations (i.e. technically true, but not what was asked). The only model I have seen that didn't at least partially hedge on this question was Grok 4, who surfaced a level of familiarity for my post history which recalled certain interactions even better than I'd remembered what I wrote.