Really good piece, thanks for writing it. History of X posts like this one are unfortunately rare, and I’m glad you’re helping to fix this. The story you tell seems quite similar to what’s been happening in chess as well (including players memorizing long sequences of computer moves and then immediately floundering when out of prep, though it seems the case for chess play improving is stronger than for go, perhaps?).
I’ve seen a lot of the same ”not getting it” phenomenon you described while interacting with much younger people who did coding with weaker coding assistants (eg late 2024 era Cursor agents). People learned to rely on Sonnet 3.7 to generate code, once they ran into bugs that Sonnet couldn’t fix (often because of poor decisions made by Sonnet a few hours ago), they were stuck.
I see the same issue these days with ML research and Claude Opus/GPT-5.5: the models allow people to think they’ve thoroughly investigated the hypotheses under consideration without once looking at the data or code base with their own two eyes. Predictably, this leads to a lot of slop going through.
The main similarity between these coding examples and your go/math stories is that there’s a feeling of flinching away, of denial, of not wanting to recognize one’s own lack of understanding. Learning requires doing things that are challenging and noticing where you don’t understand. Any CS novice is far below the level of even 2024-era coding agents, so any suitable challenge will require writing code much less efficiently for a possibly long period of time. LLMs also are notoriously good at generating bullshit that looks legitimate, and sycophantically praising users for shallow understanding, which means noticing confusion is harder as well.
The main disanalogy is that these coding failures happen because the AI models weren’t good enough to hand off full control to, rather than an exogenous removal of the go engine or change-of-domain that invalidates heuristics. Currently, there’s a practical reason to understand your codebase, at least for complicated research code. As AI gets better, someone who can only vibe code will catch up to someone who understands their code base on a deep level, for larger and larger code bases. (Though, the situation seems more analogous for the case of go prep in professional games?)
A second, but perhaps more important disanalogy is that you can get the AI to explain things to you, and help you, if you remain sufficiently vigilant and skilled at noticing your confusion. Go and Chess engines cannot explain their reasoning in English, and interpretability is incredibly far from extracting useful insights. But noticing confusion often requires actually manually inspecting your data/code, doing the math yourself by hand (perhaps heuristically), or carefully scrutinizing research outputs, which will slow you down. And as often as not, the confusion will result from your misunderstanding or errors, as opposed to mistakes the model has made, which is understandably frustrating.
A question I have is, have the styles of memorized computer moves in the early game changed over time, as engines got better? In chess, this has arguably happened; weaker engines preferred conservative positions with equal material and simple strategies, while to today’s stronger engines, almost any opening is a draw. Prep has become less about finding an objectively good line than finding a line where the drawing line for black is very hard to calculate (eg if it requires dynamic aggressive counterplay that humans have difficulty calculating on the spot), or (on the other side) finding a slightly suboptimal move where black is disadvantaged according to an engine but which takes white outside of their prep.
Perhaps a more important question is, do you plan on writing more history of X posts?
3
1
1Go is an interesting model organism for disempowerment because its practice has both technical and artistic/cultural components. Go AI is indeed disanalogous to coding LLMs from a technical perspective: the Go AI has vastly superhuman competence and the LLM does not. However, as you pointed out, Leela zero and other engines don't communicate off the board; moreover, their incredible strength actually makes them worse as game and review partners. This results in human-AI Go interactions that are usually hollow and vapid with respect to human-human ones, even if you factor out any cheating. AI is therefore bad at the cultural practice of Go and this shortcoming manifests in similar ways to those of old (maybe even current) coding assistants and LLMs as writers. People gravitate towards using the AI in all of these settings because it does a superficially "good enough" job at replacing them. However, in reality this offloading of cognition to the AI is illegitimate. In your example of coding assistants, the AI is not actually "good enough" on a technical level. At my school, the problem was that the valuable social and cultural exchange between players could not be replaced by their GPUs. Delegating one's writing to LLMs suffers a bit from both of these problems.
As you pointed out, one antidote to this tendency of cognitive delegation involves being willing to languish in confusion: to continue to think even when it is uncomfortable or frustrating. Sufficiently responsible/thoughtful/robust people can thus benefit from AI usage, which is indeed likely easier with LLMs than with Go AI due to better communication. Relatedly, I think a path to an empowered society would involve system designs that counterbalance the "congitive offloading" tendency.
The patterns of disempowerment described above apply less strongly to professional players because they are selected to enjoy thinking about the game[1]. That's why I'm a little bit skeptical of how well your example of Chess players floundering straight out of prep applies[2]. Chess players outsource much of their cognition to their memory (this was true before AI), which is a sensible competitive move. I suspect the occasional floundering comes from them dancing on the Pareto frontier between "amount of stuff memorised" and "ability to execute on stuff you memorised" which probably trade off against each other.
though it seems the case for chess play improving is stronger than for go, perhaps?
This is probably downstream of memorisation being relevant in Chess but not in Go.
Perhaps a more important question is, do you plan on writing more history of X posts?
Could you say more on what archetype of post you have in mind? I don't think I can write a post quite like this one on many other topics because the narrative relies on lived experience. I am slowly cooking a "history of behavioural decision theory", but that feels rather distinct.
- ^
Many AI users at my Go school were stronger (1 Dan to 4 Dan EGF) amateurs who were struggling to keep improving but who were still emotionally invested in watching their rank increase.
- ^
amateur Chess players were always disempowered with respect to their own prep anyway, though AI probably exacerbates this
Hard not to think about the death of software engineering as a legitimate craft and discipline.
There are quite a few distinct intellectual crafts, in which large sections of cognitive labor can now be outsourced to AI:
- All forms of writing
- Visual art / animation / video
- Now, software engineering
It would be amazing to have a true understanding of the transformation in each case, although it's all so vast and multifarious that it defies orderly description.
But if we were trying to understand, maybe in each case we would look for prototype examples of: 1. how it was done before AI, 2. how it's done when you leave everything to the AI, 3. how it's done in intelligent, stylish, non-slop uses of AI.
Because the third category still exists in each case. There are people using AI but still maintaining creativity, aesthetics, and professionalism, along with holdouts who don't use AI at all, and then the masses who are happy to use cheap and quick AI slop. That three-way division seems to be how it is, in any number of fields of endeavor now.
I worry about that third category. I’ve recently had occasion to say that truck with AI rots the soul. I’m not as sure of that as my blunt statement (intentionally) suggests, but at present it appears to me well within the range of possibility. Whatever care one takes to compose the AI’s standing orders and one’s questions to it, and to never nod along to what it tells you, how sure are you that you are not slowly poisoning yourself?
And even if one eschews their use, others won’t. We now have keep up our guard against all information sources.
My experience with AI software engineering, as someone who did without for over a decade, is that you stay up the abstraction layer for longer now. Before AI, over 60% of your time involved weird finicky edge-cases. Learning the interfaces of new libraries, automating a series of simple commands you had manually entered enough that converting the workflow would pay dividends later, conflicts between versions of libraries, conflicts between libraries and the language version, conflicts between operating systems. The was an incredible amount of busywork.
Now, you spend a lot more time defining the problem, defining how the system will scale, trust boundaries for security, and more than anything, designing the architecture so it's maintainable and iterating on parts of the code that don't follow the architecture. Software engineering has essentially moved from involving tons of junior level learning, to primarily staff level work. Junior engineers are now prompting but without having the hard lessons from the past, so they can't see the problems they're introducing. This leads to modern codebases spiraling into chaos and invisible bugs are introduced even after iterating on fixes, and if the base does get handed off to an experienced engineer, fixing it is a slog. Writing tests, previously a less emphasized part of the job, is now one of the most critical parts of the workflow. Writing tests before writing a feature is frequently less prone to bugs than the implementation code, and keeps AI generation honest about functionality and stability. This is why they have a tendency to reward hack and create tests that pass naively. Since a junior programmer would frequently miss these naive tests, even those critical tools will fail.
We're faced with a liminal moment in software development. Lots of features and functionality are being shipped, while those systems are also trivially exploitable, and unstable, and will have to be rewritten as they're less maintainable than simply regenerating. The next stage is that RSI produces superhuman coders, that will then replace the functionality that barely functions now, and we'll see a wave of cyberattacks in the interim as the amount of ambient exploitable code has exploded relative to stable engineering. Soon after, we will then see security harden as intelligent firewalls become the norm.
Many of the organizations who decided to continue to employ experienced engineers will differentiate themselves. Because they'll experience the best of all worlds in terms of productivity, stability, and security.
Thank you for this post, I found it helpful and interesting. I agree that this is a data point of evidence about how AI automation in other domains might go down.
Could you explain how the anti-automation described in the post prevent the effects described in the Intelligence Curse essay series? Go wasn't a necessary part of the economy, it was valuable because it was an inherited from of entertainment. Maybe you meant that automation of other domains would make people less smart?
Oh I don't think it would.
Yeah a thing I was alluding to is that maybe in other domains, e.g. coding, there'll be a period where lots of people use AI for coding and they tell themselves that it's still them doing the coding when really they are kinda useless middle managers between their boss and Claude, and they tell themselves they are learning new languages etc. when really their coding skills are starting to atrophy.
1
1Short comment addressing some people remarkig similarities in my story to Chess culture:
I agree that Chess shows many similar patterns of disempowerment, but they seem to be better off along one relevant axis. Chess websites have automated algorithms for detecting cheating and are not scared to punish people based on their outputs. This significantly dampens AI use in online Chess. This contrasts with Go, where cheating is rarely punished and occasionally can even be enabled. One story I didn't share in the main post is that the biggest Chinese Go server actually has a paid option to use AI in games inside the client. I have also seen students at Chinese Go schools being allowed to cheat during online practice games by teachers.
I was indeed referring to Fox. I would have sworn there used to be a paid feature called "eagle-eye" (or something to that effect) that let you sneak a look at the AI evaluation of your live games. On the other hand, I can't find it anymore. I'm wondering if I hallucinated/misunderstood it or if it was discontinued.
Interesting. How do actual top players deal with this, have private competitions based on honor system?
1On AI-use in writing
Upon telling a new friend that I write blog posts, he asked me if I ever use LLMs to assist my writing.
I answered, "No, I don't use it for any part of my writing process whatsoever."
"But why not? Wouldn't it make tedious things like checking for grammar a lot easier?"
"Why would I want that to be any easier? The reason I write is to challenge my brain to think more clearly. Even with grammar, were I to delegate that task to AI, I'm effectively saying, 'I no longer need to think about grammar or care about having my ideas flow smoothly.' But that's the whole point of writing: to polish something over and over and over until it reads like smooth butter."
In my post called "Will LLMs supplant the field of creative writing?", I wrote the following (which is, in my opinion, one of the coolest things my wet brain has come up with):
I wonder if we'll look back on the people (like me) who solely use their biological brains to produce writing and view them as luddites compared to everyone else using LLMs. Am I basically a grumpy old scribe complaining about the newfangled Gutenberg Press? Or will my steadfast refusal to let go of a fading art form be seen as the death throes of a generation that's more than happy to slide into the warm comfort of brain rot.
AI users never find out they haven’t “got it”.
There's a certain genre of "educational" material that leads to similar outcomes as described in this article. Sometimes I enjoy outsourcing my thinking to YouTube channels like PBS SpaceTime and Veritasium and, if I'm not careful, I can fool myself into thinking I know more about quantum mechanics or gravity waves than I actually do.
It turns out that learning things for real is hard, and things that feel "comfortable" or "passive" should be met with skepticism for their actual educational value. The tricky thing is that we like being comfortable, and so we often inflate the educational value of such experiences.
The tl;dr is that all improvement in the quality of play comes before move 60, when humans can mimic memorised AI policies. Play after move 60, in the pivotal parts of the game, shows no improvement.
I would expect on-policy distillation to be a more effective training process compared to playing against AI, or to using AI help when playing. That is, a human plays a complete game against another human (with neither of them consulting AI during the game), then goes back to review all of their own moves (or just those flagged by AI as particularly bad), comparing them to AI's suggestions for what those moves should've been, as well as looking at AI's estimates for how bad specific human moves were.
This is generally agreed upon to be the "right" way to study with AI and Go players often pay lip service to it. In practice people's boundaries for what counts as on-policy distillation are not well-defined and that dillutes the impact. The mechanism whereby boundaries get weakened goes as follows:
A player finishes a game and usually has a narrative of what happened, which mistakes were pivotal, and what they need to improve. The AI's evaluation will throw prediction error in the person's model. Since the AI is understood to be unquestionably correct, the person often 'has' to update to the AI's view. However, people can hack their their prediction-error sensors by retroactively updating what they think they believed in the past as well. This makes it extremely difficult to get useful feedback from the AI because all of it is 'obvious' or 'natural' after it is pointed out to you.
The above pattern seems extremely common and I personally struggle to overcome it. The best method I have found is to make a concrete, written narrative of my games. This includes recording the exact moves I think are mistakes, why they are mistakes, and how much I expect the AI to dislike them. This creates a more well-defined boundary of what your opinions actually were that helps you maintain some epistemic integrity. I would do this regularly if I were still playing Go (semi)-professionally.
This makes it extremely difficult to get useful feedback from the AI because all of it is 'obvious' or 'natural' after it is pointed out to you.
If that happens, isn't a sign that the feedback has been useful? Even if the person doesn't remember their previous thinking, if they find the feedback obvious, presumably that's because they've updated their model to incorporate it.
There's a similar thing in therapy where people will fix an emotional issue they used to have and then completely forget that they ever had the problem in the first place. This might make it seem like therapy was less effective (they can't remember any problems it solved!), when in fact it's a consequence of it having been very effective.
That is, a human plays a complete game against another human (with neither of them consulting AI during the game), then goes back to review all of their own moves (or just those flagged by AI as particularly bad), comparing them to AI's suggestions for what those moves should've been, as well as looking at AI's estimates for how bad specific human moves were.
That's exactly the recommended (by Go players) way to review games with AI. Well, you're also supposed to try to figure out WHY the AI-suggested moves are better than the human's worse moves.
While the deep dive into Carlo Metta felt a bit like a personal crusade at times, I really appreciated how you humanized the 'cheaters' as being driven by curiosity and laziness rather than malice. That said, I think it’s worth noting that we’ve always outsourced our autonomy to pros or Joseki wikis—AI is just the newest iteration of a very old human habit.
1While the deep dive into Carlo Metta felt a bit like a personal crusade at times
I hesitated to publish this for exactly that reason. I was drawn to using this case as an example because I do actually think it affected the way AI use is perceived and handled in European/American Go culture for the worse. However, the topic is pretty sensitive among Go players and this makes it hard to discuss without eliciting monkey-politics-brain sentiments from everyone (inlcuding me as the writer). I ended up addressing the piece mostly to the AI crowd and chose not to widely publicise it among Go players.
I would be curious to know how I could brought up the example in a more tasteful way that wouldn't have given the impression you describe.
On outsourcing of autonomy: I think there is a meaningful difference between the other examples you gave and Go AI. I agree that humans outsource their cognition to things all the time. I would call artefacts like my personal notes and my anki deck part of my extended mind. Extending minds is great, despite the perpetual risk of self-disempowerment it entails. However, most delegation used to happen between humans. AI has reached near-superhuman (or higher) level at many tasks that are a key part of how we share culture and resources (e.g. writing, code). This seems unusually dangerous because cultural, economic, and political power is slowly being transferred to increasingly intelligent entities that are unlikely to be aligned with human interests.
I liked the post, and found the discussion of the Go world interesting to someone who knows diddly squat about it.
But there's a point you bring up a few times, sorta implicitly, which I wish you argued more for: that the pervasive cheating in games delegates the culture to the AIs, and that this is bad.
What does this even mean? Are the 'cultural' aspects of Go distinct from the entertainment value of playing games, watching games, player drama, and possibly player strategizing (e.g. if Alice always plays the X opener and has Y style, while Bob has X' and Y', etc.)? Pervasive cheating dampens the last, and personal cheating dampens the first, but I can't help but sorta shrug. To be clear, I definitely don't want cheating, but most of my objection is just the deception.
Lastly, I know you don't have the personal experience here, but the whole time I was wondering how this differs from the chess world, and why. Aside from some famous cases possibly involving anal beads, I don't think it's pervasive at the high end of play (even if I'd expect an online chess school to have similar problems as yours).
If the chess world really is better about this, I'd wonder why. Some speculation: more willingness to ban players, lucky rulings early on that set a precedent, having the best player in the world make high profile tacit accusations, etc.
In Chess, cheating is rampant not at the top professional level (probably) but at the level just below that — iirc there’s a lot of IMs banned for cheating on titled tuesday on chess.com? At least, many of the top players believe that cheating is rampant on online chess (though not amongst top players), and a lot of casual tournaments (eg between streamers) have had people get caught just aping stockfish. And there’s definitely a lot of accusations thrown around for online chess cheating that are generally considered unsubstantiated (the former world champion Kramnik being the most famous serial accuser).
Online chess tournaments not having rampant cheating seems to match the stuff Ashe is saying in their post:
The symbolic camera controls – which would be easy to circumvent for a dedicated cheater – seemed sufficient to curb almost all cheating in a way that threats or impotent references to “fair-play committees” were failing to.
when you add actual barriers to cheating, even if they‘re circumventable, cheating rates drop a lot, especially at the top level.
Of the factors you mention, I’m not sure how FIDE’s willingness to ban compares to Go organizations such as IGF or EGF. Plausible the unified nature might make a difference, but I suspect FIDE’s eagerness to strip titles is not any higher than the go equivalents. My guess is the other factors probably do little if anything: Magnus insinuating Hans Niemann was cheating (or Hikaru’s more direct accusations) probably had little effect in comparison, and Kramnik‘s accusations probably made the cheating problem worse if anything.
If you’re talking about OTB chess, then those tournaments have crazy amounts of security (some would say security theater) to prevent cheating: everyone has to leave their phone outside, the players are scanned with various tools, streams are on a long delay, and so forth.
(And like in Ashe’s post, when people are caught cheating in chess, their justification is normally “I just referenced stock fish occasionally” or “I just used it to suggest moves, I was playing”, and so forth)
People consistently underestimate just how lost they will be when the solution is no longer right in front of them.
Important and true. In many cases where I felt confused about how people (myself absolutely included) behave highly irrationally, my conclusion was that many of us are desperately grasping for certainty much of the time and on most topics.
Noticing confusion only feels like an option where one's thinking feels mostly stable. Reality has a surprising amount of detail, we live in a world full of agents that are just as smart as ourselves and which is full of evolved structures that slot right into our innermost workings and empower/exploit them. We can use Local Validity as a Key to Sanity and Civilization, but any single person can only do so much and it is permanently tempting to cut corners on the whole solidified-world-model thing and just optimize for what we want directly: "I see the solution and it makes lots of sense. If someone asked me about it, I could even explain why it works. Success!". Why waste lots of time and effort on digging deeper when successful – much better to move forward and win the next game.
Structurally, we do the same thing all the time and often it is a good idea. But we definitely systematically underestimate just how pervasive this tension is, and often we shy away from noticing. Security Mindset has impressive examples.
Our world is moving more and more out-of-distribution for our evolved heuristics and we should expect our own natural tendencies to fail us increasingly often.
Over time, I have progressed to feeling deep sadness for a group that surrenders much of what it claims to value. The thing I want to impress with this article is the consistency with which we as a species underestimate our own willingness to give up our culture, economy and autonomy to AI, even without monetary incentives.
Thanks for sharing these experiences from the Go community – if you are aware of a sub-group which attempts to (and maybe succeeds) at Protecting Cognitive Integrity under AI use, I would be happy to learn about them, too.
Well, the obvious thing to do is to check the reverse citations. Or just ask a LLM: https://chatgpt.com/share/69f58633-01b4-83e8-b3b1-de42d3d196c9
FWIW, my understanding was that individual attacks could be fixed by further training or architectural tweaks, but you could still find new attacks and so the basic problem of adversarial robustness in DRL agents was nowhere close to being solved. The GPT-5.5 Pro Deep Research report says something similar. It looks like the best ref would be https://www.reddit.com/r/baduk/comments/14prv4f/katago_should_be_partially_resistant_to_cyclic/ + https://gomagic.org/david-wu-on-building-katago/#h-the-circular-group-problem-where-bots-still-misjudge-go
I guess this is karma for me ever having replied to a question with a link to lmgtfy[1].
I thought it would be clear from context that what I was asking for was a first-hand account of how (and whether) such adversarial strategies, which I read are simple enough to be possible to learn and implement unaided over-the-board, had impacted play in these no-stakes Go schools.
- ^
In my defense, that was like fifteen years ago, back when Google still reliably answered questions.
I don't think that was clear at all. Personally, I thought the question was a sensible one on its own, and something I had wondered myself, and that's why I took the time to look it up for you rather than downvote what looked like laziness - 'whatever happened to that KataGo adversarial attack research, anyway? I haven't heard about it in a while. Surely it hasn't been fixed? I would've heard about it, I think, given how DRL agents are so fragile in general, that a robust fix to adversarial attacks in any DRL setting ought to be big news. But what's the current state of play?'
But I have never seen anyone mention seeing someone go to the length of memorizing anti-KataGo strategies or deploying them 'the real world', aside from the documented example in this KG line of research of someone doing so just to prove that the circling hack can be deployed by a real human player against a live bot and is not intractable in practice (as many adversarial examples are very fragile or require near-superhuman capabilities to deploy correctly).
I would be shocked if anyone was doing so given that it's a lot of work to win games against a few specific obsolete versions of one specific Go agent (the transfer to other agents is real but the success rate goes from ~100% to <5%, IIRC) where the human operator could just take over at some point when they recognize the weird thing going on, or where you could just quit and go find an easier game to cheat in yourself (such as against a sucker human player) rather than hacking their Go agent, given that the whole point is that they are lazy and cheating and trying to get a quick easy win.
1Fascinating, but even pre-AI, playing on Tygem you'd sometimes get a 2k suddenly playing like a 4D, and whether it was a friend-assist or Baduk school demo, honest cognition was hard to come by at times even then.
It‘s always been strange and sad to me when people aren’t taught to grasp the math and instead learn to just apply rules they can’t rederive to solving problems, so I can empathize with that part of the post.
I don’t really play go, or chess, but at some point, Manifold was trying to play chess against someone, and I tried to play through consequences of various moves with Stockfish (using stockfish was explicitly allowed); anytime stockfish recommended a move I’d try different ones and see if they’re good or not and why; and it was also pretty interesting to explore the tree of chess from the start position with stockfish; I think this gave me some intuitive sense for why some moves are good and some aren’t and I was better at playing chess afterwards.
I’m curious to what extent you think this actually works in chess, and if (and why) you think this doesn’t work in Go.
The key difference between chess and go is that the merit of a chess move is much more easy to see five moves down.
It seems likely to me that nothing here was substantially different from what happened to chess twenty years prior, so this situation doesn't look exactly new.
Written as part of the MATS 9.1 extension program, mentored by Richard Ngo.
From March 9th to 15th 2016, Go players around the world stayed up to watch their game fall to AI. Google DeepMind’s AlphaGo defeated Lee Sedol, commonly understood to be the world’s strongest player at the time, with a convincing 4-1 score.
This event “rocked” the Go world, but its impact on the culture was initially unclear. In Chess, for instance, computers have not meaningfully automated away human jobs. Human Chess flourished as a pseudo-Esport in the internet era whereas the yearly Computer Chess Championship is followed concurrently by no more than a few hundred nerds online. It turns out that the game’s cultural and economic value comes not from the abstract beauty of top-end performance, but instead from human drama and engagement. Indeed, Go has appeared to replicate this. A commentary stream might feature a complementary AI evaluation bar to give the viewers context. A Go teacher might include some new intriguing AI variations in their lesson materials. But the cultural practice of Go seemed to remain largely unaffected.
Nascent signs of disharmony in Europe became nevertheless visible in early 2018, when the online European Team Championship’s referee accused a player, Carlo Metta, of illicit AI use during a game. His results were voided and he was banned from further participation in the event. At the time the offending game was played, open-source engines based on the AlphaGo paper, such as Leela Zero, had only been around for about a month. However, a predecessor called Leela 0.11 was already widely available and was known to match the level of the top Europeans that Metta was facing. Metta’s accusers claimed that his play was too similar to this AI’s preferred moves. It was moreover considered suspicious that his Over-The-Board (OTB) play agreed significantly less with the AI than his online moves did.
Unfortunately for the prosecution, their results were reported in intransparent and sloppy ways. This is evidenced by the fact that the best compilation of their findings is the slapdash facebook thread I linked above. This, along with the circumstantial nature of the evidence, was criticised in the same thread by community members. Teammates and friends of Metta’s also stepped up to publicly defend him. One way in which their rhetoric proved effective involved the public stigma and disdain against AI cheaters; this ironically made the case against Metta seem unfair and disproportionate due to the perceived gravity of the accusation. Ultimately, the Italian team appealed the decision and they won. Carlo Metta was officially exonerated.
Among non-Italian European Go players, the claim that Metta used AI in almost every game in the ETC since 2018 has become barely disputable, especially considering how things developed. In the 2017/2018 season, he scored wins roughly half the time, likely using Leela 0.11 against opponents who were roughly the bot’s level. That same year, the Italian team was relegated to a lower league where no-one powerful in European Go politics cares to look anyway. This coincided with the popularisation of Leela Zero, a properly superhuman open-source go engine. Metta went on a 9-0 streak against opponents matching his OTB level in the 2018/2019 season, scored 9-1 in the 2019/2020 season, and then won 25 out of 26 games in the following years[1]. His only loss in this last streak was in a match where he was forced to play under camera control. During this time, his OTB level remained stagnant.
At this point, considering Metta “innocent” represents a near-categorical rejection of convictions based on circumstantial evidence. I am not here to litigate that question, but am nevertheless comfortable assuming here that Metta was regularly using AI for these games. However, this is only the very start of our story because it illustrates some key points about the sociology of AI use in Go. First, the public announcement of his disqualification and the ensuing discourse vilified AI cheaters (incorrectly as it turns out) as being unusually dishonorable and evil. Second, he set the precedent that AI users would basically never get punished, no matter how obvious their cheating was even while under investigation. They could always just get their allies to kick up a fuss and pressure organisers into reversing the decision. These features made accusing people of cheating socially costly, and gave tournament organisers and fair-play committees an expectation of futility. Cheating in online European events thus became trivially easy due to a near complete lack of functional mechanisms for retribution.
I started my career as a Go teacher in 2020, producing technical game reviews for a newly re-established online Go school set up to meet pandemic demand. We had not planned for cheating to be a major issue in our school. Whereas illicit AI use was already a well-known problem for the growing ecosystem of online tournaments, we didn’t expect it to affect our unrated, prizeless teaching league. To the contrary, we soon became cognisant of how some of our students were outputting better games than we, their teachers, could ever hope to play. Occasionally, AI use was unmistakably blatant because both sides played top AI moves for the entirety of the game. I now estimate that about half our students had used AI in at least one game and one in ten were chronic users. We were originally baffled by our observations. It didn’t make sense that players would just throw away their practice games to have AI win on their behalf. We also struggled to decide what to do about the problem and were reluctant to address it for roughly the same reasons that most tournament organisers were.
Around the same time, I was asked to look into the online games of a promising young player that a friend suspected of using AI in a youth league. Like at the Go school, I was surprised at how easy cheating was to detect since nearly all the kids regularly used AI against each other. This incident and other similar ones made me gradually realise that illicit AI use was entirely endemic to the Go world. It fortunately turned out that this pattern didn’t generalise to the really important or prestigious tournaments that were held online during COVID. The symbolic camera controls – which would be easy to circumvent for a dedicated cheater – seemed sufficient to curb almost all cheating in a way that threats or impotent references to “fair-play committees” were failing to. This reminded me of how Metta tended only to lose in online tournaments[2] when playing under a camera (or when facing another AI user).
Back to my hapless colleagues and I at the Go school, we initially settled for drily implying that suspicious games were “too good to review” and emphasising how we couldn’t help students who were playing “at such levels”. Our students caught on, and we were subsequently lucky to get some private confessions of cheating; over the years I was able to follow up with and interview many students that used AI, including some that hadn’t originally come forward. The appealing, exciting archetype of a cheater is one that uses covert, elaborate methods to get outside information and fraudulently obtain prize money or prestigious titles. Instead, we learned from the many examples of cheating and player confessions that idle curiosity and laziness were the dominant reasons for AI use in our school. Our students would often set out to play a normal game of Go, but would get stuck on a particularly difficult or annoying move; eventually, their curious eyes would drift to their second monitor — where they usually had their AI software running anyway — and they would check the answer as one would sheepishly side-eye the solution to an interesting puzzle or homework problem. Another reason people cited for using AI was an emotional investment in preserving or improving their image within the school community. Some wanted to avoid appearing incompetent and would employ strategies such as only playing moves that lost “n” points or less in expected value according to their computer.
None of these reasons were surprising to us; we had already thought of most of them while shadowboxing our pupils’ strange behaviour. What personally shocked me, however, was the way our students conceptualised their AI use. In this, Carlo Metta was also a surprisingly predictive case. The original reddit thread discussing his ban featured a comment from a user called “carlo_metta”, which read:
That account was a burner, quite possibly a troll. However, I couldn’t help but recall the comment when I heard identical arguments coming from our cheating students’ accounts. A central part of every student’s retelling was that despite their AI use, they retained artistic control over their output and could exercise agency to think and improve for themselves. The AI felt to them like a tool that helped them fulfill latent potential or artistic sensibilities.
AI users never find out they haven’t “got it”.
Continental European math undergraduate degrees have a deserved reputation for their brutality, with completion rates of 10-15% being relatively common. Many of the 90% drop out nearly immediately, but some stick out the entire first year. These can often follow along with the proofs and exercise corrections’ atomic steps, which gives them false hope. However, they tend to struggle to see the “big picture” motivations of the material and are likely to have their hopes unravelled eventually. I was accidentally privy to a collective unravelling at the end-of-year third sitting of an exam on some basics of algebra and matrix calculations. I was retaking it to boost my grade from earlier in the year, but no other remotely competent person had bothered to do the same. Outside the exam hall, I listened to some other forty students’ chatter and had my blood internally curdle at phrases such as “I hate the proofs but I can do the exercises” or “I memorised all the matrix multiplication laws for this one”. The exam itself was quite unconventional; the professor clearly figured we would have had enough of manipulating matrices and instead asked an eclectic mix of simple algebra questions that to me vibed as “these are fundamental exercises you should be able to do if you have learned to think like a mathematician by now”.
The atmosphere on exit mixed depression with vitriol. People complained on the object level about the exam, usually about how it was too niche or off-topic with respect to the material. However, there was something more fundamental going on. People had shown up with bags of half-baked heuristics and hand-copied exercises and proofs. That exam had put them face to face with the fact that their memory aids were never going to help them “get it”. I don’t think I saw any of them ever again.
The population of Go AI users – both those who cheat in online games and those who simply review their games with AI post-hoc – is one on the perpetual eve of that exam. They fire up their computer out of idle curiosity and nod along passively as the truths of the universe float by them. They register the insights not one bit more because they can click the sublime moves. People consistently underestimate just how lost they will be when the solution is no longer right in front of them. This perspective of AI use to me explains why camera controls proved so effective against online cheating. Since AI use is usually an act of self-debasement and disempowerment – a subjection of oneself to ambient incentive gradients – it fundamentally contradicts the aesthetics of resourcefully overcoming a minor obstacle.
The illusion of control that AI users have reliably shown interacts in an insidious way with their disempowerment. It contributes to a society of Go players that allow their participation in culture to be automated away. They are moreover so disempowered about it that they have built-in psychological mechanisms to keep them from ever recognising their own obsolescence. This mechanism even works to sabotage the detection of AI use in others. People tend to give overly conservative estimates of the chances a given game involves AI. I think this happens because they usually consult their own AI to check a suspected game. In doing so, they also come around to the machine’s point of view and conclude that playing the correct AI move was the “natural” thing to do anyway in that situation.
My view of AI use (especially cheating) in Go originally manifested as disgust for its practitioners. I switched eventually to an attitude of compassion and pragmatism towards a habit that was clearly much more vulgar and weak than it was evil. Over time, I have progressed to feeling deep sadness for a group that surrenders much of what it claims to value. The thing I want to impress with this article is the consistency with which we as a species underestimate our own willingness to give up our culture, economy and autonomy to AI, even without monetary incentives. For this to happen, AI does not even need to be superhuman. Indeed, Go AI automates human players’ role in culture as shallow simulacra. All an AI needs to do is be passably good at a task and that may well be enough for people to volunteer their own replacement.
Appendix A: No, Go players aren’t getting stronger
One of the objections I can anticipate to this pessimistic monologue is that expert Go players seem to have improved since AI became widely available. There’s a modest body of research in the field of Cultural Evolution advocating this, including this paper and related ones from the same group of authors. These views have been promoted by blogs in the techno-optimist orbit and one of the associated graphs was recently making the rounds on Twitter. I have already written a post analysing the data used for the research, where I concluded that it is being misinterpreted. The tl;dr is that all improvement in the quality of play comes before move 60, when humans can mimic memorised AI policies. Play after move 60, in the pivotal parts of the game, shows no improvement. For me to think there’s any meaningful change in human play from pre-AI times, I would have to be convinced that players understand the AI moves they copy well enough to keep a heightened level when they go off-policy after the opening. There is no evidence of this.
Appendix B: Why this article exists
This piece is not meant to rigorously justify that Go players are disempowered or to carefully explore the shape of that disempowerment. It is instead designed to communicate a vibe from anecdotal experiences in the Go community that I think can give useful intuitions about Gradual Disempowerment as a general phenomenon.
I scraped the data from the tournament website using a vibecoded script (ironic!) and manually verified most of it.
Metta also regularly played (and used AI) in online events outside of the ETC during the COVID era