On Positionality: I am a non-Chinese person; I don’t speak Mandarin, and I have not lived in China before. I have written this piece after reviewing English publications on Chinese politics and AI-enabled extreme power concentration risk for ~12 weeks. I tried to account for my detachment by requesting feedback from people with China expertise.
Summary
I define Extreme Power Concentration (EPC) here as economic and coercive power being concentrated in the hands of the few who control the most powerful AI. The emphasis on the few is very important because the number of actors who end up in power differentiates between a world where power is “extremely concentrated” and another where power is "concentrated." Despite the recent attention to the AI-enabled extreme power concentration problem, little work has investigated what would happen if China-the first runner-up in the AI race happens to develop AGI first. The disparity between the American political system and the Chinese political system further limits the transferability of the current threat models and interventions to China and thus makes us less prepared for the manifestation of CCP-led EPC risk.
Through this project, I tried to assess whether the theoretical argument of AGI enabling ~ one person or an extremely small group to independently run an entire country (in this case, China) is plausible given the seemingly high propensity for one-man rule in China, and if so, what can we do about it. I start from the assumption that China managed somehow to secure the lead in the AI race over the US and that it solved its compute bottleneck and achieved chip sovereignty. Therefore, model distillation pathways won’t be sufficient for this scenario[^1].
My thesis is that while AI would make power more concentrated[^2] in China, extreme power concentration (EPC) is hard to materialize in the CCP. I argue that in a Fast EPC scenario, the power seeker would need buy-in from other actors in the AI tech stack and CCP officials to make it happen. Therefore, incentivising those actors (to create a world in which they won’t be needed anymore) would break the main assumption EPC relies on (that power seeker(s) can do things without other humans if they wanted to), given that these other actors realize the power seeker(s) plan and demand sustained leverage. This makes it hard for the power seeker to realize EPC with a very small number of actors. That said, it can be possible to realize Fast EPC with a relatively large coalition of actors governed by a power-sharing agreement.
In a gradual EPC scenario, cooperation for EPC would be a survival strategy and effectively the norm, and other actors may not realize the world they are heading towards; therefore, their power would decrease gradually. However, given the US lead in the AI race dynamics between the US and China, we are more likely to live in a Fast EPC scenario than a Gradual EPC scenario because I assume that China must build AGI first to realize EPC[^3]. Also, EPC is more likely under Xi’s term than under his successors due to the personal loyalty Xi has managed to build over the years. This loyalty would make it easier for him to realize extreme power concentration conditions.
The main conclusion of the piece is that we should talk more about EPC to help both Chinese actors and US policymakers internalize it and develop a healthy paranoia toward AI acceleration in China. We should think about potential interventions for this risk despite its low probability because some of the interventions could benefit less extreme scenarios of the risk which are more likely. Particularly, I recommend:
Continue to apply pressure on the US to win the race[^4] through advocacy, lobbying, and engagement with think tanks that already think about China (e.g. CSET, RAND).
Raise awareness about extreme power concentration risk in China through:
a. Including EPC risk in Track 2 dialogue with Chinese academics and think tank experts.
b. Conversations with organizations in the AI safety space, e.g. Concordia AI and Open Community for AI Safety China, about EPC risk.
c. Utilizing Chinese networks of:
i. international organizations working in human rights and journalism in China (e.g. Human Rights Watch, The Economist, Freedom House)
ii. Chinese overseas.
iii. regional networks in the AI safety space and AI safety hubs in Hong Kong, Taiwan, and Singapore.
d. Indirect measures like supporting Open Community for AI Safety China in building awareness of broader AI safety concerns can help in making Chinese actors more skeptical toward AI acceleration in China.
Strategically reframe concrete technical safeguards for EPC to align with CCP AI safety priorities (AI misuse, AI loss of Control).
Invest more in China specialists in AI safety fellowships (IAPS, GovAI, MATS, etc.) and China projects in AI safety RFPs to expand the circle of people thinking about extreme power concentration risks in China.
The piece starts with a detailed EPC scenario to help readers internalize what this EPC world may look like before discussing how EPC would change dynamics within the CCP, conditions for realizing EPC, potential models for EPC, barriers to EPC, and interventions for EPC.
1. What does this EPC world look like?
Note: this scenario is an attempt to help the readers picture the worst-case version of the extreme power concentration risk under CCP. It’s not a forecast, so if some of the assumptions here seem implausible, feel free to skip ahead; the arguments in the rest of the piece do not depend on the speculative details here.
Imagine this: we are in 2035, and Xi Jinping is still alive. In the last ten years, China launched its AI++ plan, made unexpected progress in its chip industry, and managed to reach Artificial General Intelligence (AGI), which surpasses human capacity in all tasks. The AGI robots can do anything autonomously. They can draft CCP governance plans for the 1.41 billion people, implement them at the local level, and report the progress to Xi. This is not specific to economic functions but also to national security functions. AI has been integrated into the People's Liberation Army (PLA) with fully autonomous weapons and AI strategic planners that made an astonishing victory in Taiwan, and the rest of the world, including the US, either surrendered or was defeated and became part of the greatest China.
1.1 What does this world look like for CCP members?
This means that for most of the 101 million party members, AGI has taken their job[^5]. The party members did not expect this because they were optimizing for AGI development, and there was some sort of pact that they would maintain their jobs after integrating AI advisors[^6]. It was difficult to predict this because conversations on AI risks were censored, and party propaganda made it seem like the CCP has control of AI risks and that AGI will serve socialism with Chinese characteristics and the great rejuvenation of the Chinese nation.
1.2 What does this world look like for Chinese citizens?
Xi continued to oppose welfarism. He thinks people should not be granted wealth without working for it. So he invented jobs (similar to inmate work programs[^7]) for humans as a way of disciplining themselves and also as a condition to qualify for universal basic income (UBI). These jobs include studying and writing extensively on Xi Jinping thought[^8]. Other jobs include teaching, art production, and nursing[^9]. If you started thinking this future is not too bad, just consider the fact that you won’t get a chance to choose what your job is, and you basically can’t quit. There is no attractive future trajectory that you can compete for (one in which you would get a better quality of life if you have worked hard for it). Also, ordinary citizens do not have any guarantees that Xi won’t wake up one day and decide to cut off all the UBI, apart from good faith in Xi’s moral compass. In this world, all people get the UBI amount determined by the AGI. As far as we know, the criteria it considers to determine UBI are whether a person:
Worked in the jobs Xi invented (positive correlation)
Loyalty screening results and your credits in the social credit system. (positive correlation).
Need-based system (health records, number of family members, etc)
The CCP’s definition of an ideal Chinese citizen (beliefs, social attitude, etc.) will be perfectly enforced. If you feel like this is already happening in China, imagine the enforcement power AGI can bring. Even for choices like marriage, people would have to do it the CCP way; the AGI matching algorithm would pick their potential human/AI partner and later the eligibility for having kids. The AGI would be the CCP’s “angel on the shoulder”, guiding you in every social situation to behave in a way that Xi endorses.
All people will be made atheists by talking for hours to a super-persuasive AI that is trained on party propaganda, and there would be no need to manage minorities like Uyghurs, or Tibetans because everyone is part of the majority engineered by the CCP. There is no other place to escape to because Xi has power over all countries and even space.
You can’t pretend in this world because the surveillance is stronger than ever; using brain signals, Xi can get access to people's ideas and feelings even if they did not express them.
For those who resist, they get into a brain simulation equivalent to an “experience machine,” but one that answers for Xi only, and it optimizes for suffering and pain for his defectors. In this machine, you would feel all the horrible things a person can feel during multi-round simulations (death of your loved ones, painful disease, starvation, physical and sexual violence). Compared to the CCP experience machine, the horrible suffering currently happening at the CCP detention camp would be considered merciful.
1.3 What does this world look like for non-Chinese citizens?
For those who are non-Chinese, the situation is even more difficult. Under the unification vision, Xi Jinping is turning the whole world into the greatest China. Foreigners are expected to immerse themselves in Chinese culture and values through talking for fixed hours with superpersuasive AIs[^10].
After being cleared from the loyalty screening, foreigners are eligible for half of what UBI recipients in mainland China get, and as an act of revenge, anyone who ever shared negative views about the CCP before would experience the suffering machine (this piece would put me in a disadvantaged position in such a world).
1.4 What does this world look like for AI?[^11]
Xi transferred most of his propaganda techniques to the AI alignment process, which resulted in AIs that take Xi’s thought as ground truth and are internally motivated to advance Xi’s interest. While he resisted AI welfare policies at the beginning, he slowly extended his views on welfarism to AI, meaning that welfare won’t be granted to AIs unless the AI proves its worth and loyalty to Xi.
1.5 On the possibility of transition
The main issue is that there is no way for anyone to change this state of society. Value is locked in, and whatever Xi sees as best will continue to determine what’s best for everyone forever.[^12] Xi never dies in this world because his new AGI-led health preservation center discovered effective drugs for immortality, so his term effectively becomes unlimited[^13].
2. How AGI May Change Power Dynamics Within CCP?
Before discussing the likelihood of this EPC world, I want to make the case that if it happened, it would change the power dynamics not only between the CCP and the public but also within the CCP.
My initial question about AI-enabled power concentration in China is, "Isn't power already concentrated in China?" The answer is yes: power in China is already concentrated, but AGI will make this worse. Contrary to my initial assumptions, there are dynamics inside the CCP that decentralize power “to some extent,” and conditional on the likelihood of EPC, AGI could remove them. Below, I try to analyze the current decentralization dynamics and how AI may change them.
I argue that AGI will remove the delegation dynamics within the CCP, as AIs replace CCP officials. Below, I list the reasons why Xi would be convinced to replace CCP officials with AGI:
The AI will be better, faster, and more efficient than CCP officials in performing governance tasks.
The durability of AI loyalty is what sets it apart from human loyalty. Once we figure out the technicalities of AI alignment, AIs will (at least theoretically) remain loyal forever.
The AI will not have empathy for a group of people like local governors would for people they have human relationships with or because of some moral awakening.
The AI will not optimize for its own interests (accumulate resources and engage in corruption).
2.2 Competition
The CCP uses competition to incentivize delegates and limit the power of each individual actor, which in turn helps the CCP achieve its governance objectives. The most obvious one is among Chinese businessmen/businesswomen, but the impact of political competition among Xi’s subordinates is often overlooked. The competition between CCP elites over promotion and being Xi’s favorite helped them check each other's power. The competition is a prevalent theme in navigating the job market and the exams in China. So whether the CCP manufactured competition or it happened systematically as a coping mechanism, it tells us that as long as there is a chance for people to “play the game," people are channeling their energy to win by the rules of the game set out by the CCP rather than attempting to change the rules of the game. I argue that, absent countermeasures, AGI may break the social contract for party members and ordinary citizens, where loyal officials who play by the rules of the game won’t get a chance to climb the party ladder or accumulate resources. For citizens, it means that perfect obedience and hard work will not be rewarded (In my scenario above, Xi mitigated this by using UBI as a reward mechanism).
2.3 Information asymmetry
There are logistical barriers to telling superiors everything, and the discrete judgment of deciding what’s important to tell decentralizes power.[^14] AI will remove mechanisms because it will unlock perfect surveillance and centralized processing of information.
2.4 Death[^15]
Although in 2022 Xi Jinping broke the term limit rule[^16], death continues to be an uncontrollable limit for Xi’s power. The impression is that Xi would opt for a fourth term and continue to rule until he dies[^17], similar to Mao, who served as party chief until he died at the age of 82. AGI may change this by unlocking unlimited terms either through new drug discovery breakthroughs in longevity science[^18], or through creating a digital twin of Xi Jinping, an AGI that is fully aligned with Xi’s thoughts.
All of this motivates me to think that AGI would be an attractive asset for power seeker(s), and it would definitely make power concentration worse inside the CCP.
Now that I have established how AI will change the dynamics inside the CCP, I move to explain my best guesses for how we would move to the EPC world through what I call conditions for the EPC before sharing my estimates on its feasibility.
3. Conditions for EPC
There are several technical and non-technical conditions the power seeker(s) will seek to control to achieve their EPC vision. Ideally, the power seeker(s) will attempt to control all of these conditions to eliminate their dependency on others, but realistically, and depending on the speed of the takeoff, they would end up in a world where they get buy-in from others, who in turn control the conditions via a power-sharing agreement. Below, I classify them into technical, institutional, and technical governance conditions.
3.1 Technical Conditions
The major condition for EPC scenarios is the ability to develop AGI[^19]. There is limited consensus on whether it’s possible to build AI that matches or surpasses human intelligence. If humans managed to do so, we would probably start from AI R&D where the AIs recursively build future AIs, leading to an industrial explosion that fully automates the economy.
Alignment target: I define alignment here as teaching/instilling the values and preferences of the power seeker(s) in the model. This is similar to “loyalty” in a sense where the power seeker(s) may want the AI models to optimize for their interests against their own interests (self-preservation, power seeking, etc.).
These conditions are mainly controlled by people with technical expertise or those who have control over them. These technical experts may include developers in the AI labs and academics at leading universities in AI research (e.g., Tsinghua University, Peking University, etc.).
3.2 Institutional Conditions for EPC
Direction: What Gets Built
Control of the plans that set the road for what gets built through:
National plans (party, State Council and National Development and Reform Commission (NDRC)).
Requests for Proposals (RFPs) from state-owned Enterprises (SOEs) and PLA.
Control over industry norms through standard-setting: bodies like TC260 issue technical standards that inform practices in the field.
Control over ethics boards' review mechanisms that approve proposals for AI R&D models before development.
Resources
Control over the AI tech stack: decision makers at the AI tech stack (funds, energy, minerals, compute, data) who decide how to allocate resources[^20]. This condition is more decentralized and ambiguous than other conditions, but it’s essential for control over AI functions. In the Fast EPC section below, I attempt to unpack this and name a few actors.
Control over the information flow of what’s being built (security apparatus). Also, the security apparatus will play a huge role in safeguarding the power seeker(s), as it is in control of coercive power.
Access controls of the model: which involve control of the model use (developers control) and controls over the model possession (open-source vs closed-source weights and who can download the open weights) and its exports.
Post-deployment monitoring: currently, developers are required to monitor their models after deployment, and CAC runs periodic enforcement campaigns. I expect this to be an important requirement only if stronger enforcement mechanisms were introduced and the scope of the monitoring expanded to include internal deployment.
3.3 Technical Governance Condition
Evaluation & Verification: Being able to verify that the model is aligned would be an important condition for EPC. Actors in CAC and MIIT-affiliated bodies (CAICT, CESI, CEPREI), Shanghai AI Lab, and ethics review boards would have leverage over power seeker(s) because they would enable the power seeker(s) to:
Ensure that the model is aligned with the power seeker(s). This includes:
Fabricate reports and facilitate permissions to build and distribute the EPC model to avoid being caught by members outside of the coalition.
That said, this evaluation knowledge (without other conditions) won’t be sufficient for EPC. Also, there are two possibilities under which the bargaining power of these evaluators may become smaller if:
Their influence ends as soon as AI judges handle the verification and evaluation for future models (especially since scalable oversight is getting more popular in the field[^21], so there wouldn’t be cultural resistance in the AI space to it).
3.4 Why is EPC led by leaders at AI labs alone not feasible?
The direction, verification, and deployment conditions are where Xi and party members would have the most leverage over actors in the private sector. This is because, although in practice private actors have leverage on development and they can build models that are secretly loyal to them, they would struggle to arrange this under the CCP surveillance machine. And even if they coordinated that outside China, they would need actors in the CCP to cooperate with them by:
Allowing them to build the risky models (e.g., ethics boards approve their AI R&D proposal)
Allowing them to acquire more resources in the AI tech stack (e.g., assigning more compute for their lab)
Turning a blind eye to the flaws in their verification after development.
And while, in theory, these businessmen/businesswomen can use the race to legitimize cutting corners to get buy-in from some actors, there is no guarantee that CCP officials would help them at all times over their competitors; also, the lack of incentives for these CCP officials would stand in the way of getting things done quickly and efficiently, a prerequisite for Fast EPC. This motivates me to think businessmen/businesswomen led EPC in China is hard to achieve.
3.5 Why is EPC Without Xi not as likely?
I argue that if Xi dies, his successor would find it harder to concentrate power via AI because it takes time to build personal loyalty and tune the institutions in a way that serves the power seeker's interests (consider how Xi in his first term was relatively weaker than his current version in the third term). Moreover, it seems like power concentration in the CCP is not only about the post but also about actors’ agency and political skills, which influence the space of possibilities. For example, Xi Jinping was able to concentrate power (via removal of factions and the second-term limit) in a way his predecessor (Hu Jintao) could not. My take is that depending on the speed of the scenario, if we end up in a CCP EPC world, it will probably be led by Xi alone or Xi along with other actors in the AI tech stack, as I will explain in the upcoming section.
4. Modeling Potential Scenarios
I consider two scenarios for EPC. Below, I explain their implementation, prerequisites, consequences, arguments against their likelihood, and tentative signposts to watch for.
Scenario 1: Gradual EPC
Scenario 2: Fast EPC
Speed of the takeoff
Gradual and relatively slow
Fast
Actor(s) that end up with power
Xi Jinping alone
Coalition of actors
Other actors’ leverage
Low because their cooperation gradually becomes the norm.
High because their fast cooperation is needed.
What is needed for this scenario?
Diplomacy or technical innovation to block the US from reaching AGI first. Strengthen evaluation capabilities for secret loyalties and evals gaming.
China builds chip sovereignty. Incentives for actors to accelerate development.
Arguments against its likelihood
Stalling the US from building AGI is unlikely to succeed for long and the US is more likely to build AGI first.
Fast EPC is hard because a large number of actors may not cooperate without sustained leverage. It may be technically infeasible to build chip sovereignty fast
The increased Chinese investment in chip sovereignty. AI acceleration continues.
The distinction is important because actors’ leverage, power distribution, and therefore entry points for interventions would be different under each scenario. Below, I try to explain each scenario in more detail.
All the conditions I discussed above are somewhat controlled by the CCP, so apart from creating strict AI evaluation to tighten party control, nothing has to change to realize gradual EPC. The way I imagine this to happen is as follows: extreme party control and the competition between labs would create a race-to-the-top dynamic in which AI labs optimize for alignment with Xi, slowly moving towards AI henchmen without realizing it. It can be strategic or systemic. Similar in speed to gradual disempowerment scenarios.
In the gradual EPC, the compute and robotics infrastructure would be gradually built by humans at the same time we are accelerating AI development, and when we reach software-only AGI, we would have most of the infrastructure needed to automate the remaining parts, so there would be no single point in the future at which we would start building the infrastructure for industrial explosion and military AI.
4.1.1 Why Would AI Companies Cooperate in the Gradual Scenario?
The initial question I had is, why would AI companies and other actors do that? Why wouldn't they bargain to sustain their leverage in this gradual EPC world?
The answer to this can be inferred from the censorship analogy. Aligning AI to Xi’s thoughts and perfect obedience to Xi can be implemented in a way that is similar to censorship, where the social media companies just do what the party wants, or else they would be shut down. I argue that the key aspect for strong enforcement is the competition and the delegation of censorship to the companies[^22]. Beyond AI-driven capacity gains, time has also significantly strengthened the censorship machine, as practices evolved from simple to sophisticated approaches that are hard for citizens to game, and, most importantly, time helped the public and the social media companies normalize censorship. This is the exact mechanism I am imagining to see with AI alignment and evaluations, where Chinese institutional genes will pave the path towards AI-enabled extreme power concentration; this will happen gradually by triggering a race to the top between Chinese labs to ensure their models are fully aligned with Xi. Current evaluation dynamics already replicate the censorship dynamics where the responsibility of running evaluation (e.g. filing for CAC review and ethics boards) is mostly delegated to the AI labs.
The leverage other actors would have in this gradual EPC scenario is limited because, as in censorship, the gradual nature makes it easy to normalize it. Also, the incentive structure will reflect this and reward cooperation in the short term. People would not internalize the risks because they would be so busy with the short-term incentives of cooperation (getting promotions and funds for accelerating EPC). Cooperating with Xi to realize EPC would be the new normal stable state of Chinese society; therefore, collective action (among actors in the private sector, CCP officials, or the public) to change this would become hard to trigger, and Xi would have enough time to strategize about EPC (for example, introduce UBI before Chinese people fully internalize the risk of job displacement).
Interestingly, the task of political evaluation of public-facing AI models initially was similar to censorship with a keyword list to control what the model outputs to the public, but now, as the shift has transitioned from what the AI says to what the AI does and values internally, the task of evaluation with Xi’s values to achieve Gradual EPC is likely to be bottlenecked by technical gaps rather than political will within China, as we will explore in the barriers section.
4.1.2 Gradual EPC and the US lead
The main blocker for Gradual EPC is whether the US built AGI first, because it’s not clear that if the US reached AGI first, it would not interfere with China. After all, they would be too powerful, and there is no global government to check their power. To that extent, China's current efforts to call for global AI governance may be seen as a defensive move to ensure nations' sovereignty in the post-AGI world. This can enable Gradual EPC in China even if the US reached AGI first, but I am not confident that this is feasible. An easier strategy for Chinese power seeker(s) may be to use current Trump-Xi dialogue momentum to convince the US to slow down because of false positive loss-of-control risks in the AI notification mechanism while China secretly works to bridge the gap.
4.2 Fast EPC
While gradual EPC is the default pathway, its likelihood is undermined by the possibility of the US winning the race first. Therefore, under AI race dynamics, the power seeker(s) may attempt to speed up EPC.
The way I imagine this to happen is that Xi would incentivize actors in the AI tech stack to cooperate and build AGI fast to achieve the great rejuvenation of the Chinese nation. Similar in speed to AI-enabled coup scenario. After achieving chip sovereignty and reaching software-only AGI, human workers would be tasked with the physical labour of building the compute and the robotics infrastructure to facilitate full automation of the economy and military AI.
I argue that the most likely scenario for a fast EPC is a coalition of actors rather than a single actorbecause actors would require incentives to work toward realizing Fast EPC. And while negative incentives can be easier to introduce, they result in more risk-averse behaviour, the opposite attitude for AI acceleration and development. Also, unlike a gradual scenario, the power seeker(s) don’t have the privilege of time to strategize and tune institutions gradually; therefore, other actors (developers, factory workers, etc.) would have more leverage than in the gradual EPC scenario. To secure buy-in, the power seeker(s) will probably start with political mobilization and manipulation using the Chinese rejuvenation slogans to incentivize actors first before proposing a share in the AI power because It’s in the power seeker(s)’ best interest to try to minimize the leverage other actors have in the EPC world. Once this strategy fails with some actors, depending on their leverage, the power seeker(s) may propose a power-sharing agreement and offer them:
Control over shutdown buttons: highest leverage, e.g. Xi Jinping alone
Have a say in the alignment process (by ensuring their values are represented in the Constitution or Model Spec for the RLAIF process): high leverage, e.g., irreplaceable actors in the AI tech stack.
Access to run an instance of the AGI model: Medium leverage, e.g., evaluators
Share of AGI-generated wealth: low leverage, e.g. factory workers.
Also, for each condition, only actors at higher layers (top officials, CEOs) would likely be included in the coalition, whereas lower layers (AI engineers, builders of infrastructure, junior CCP officials) would not be given high leverage unless their manipulation by their leader is not possible and they are irreplaceable.
Below, I try to list potential actors in the Fast EPC coalition with my guesses on how replaceable they would be for the Fast EPC scenario.
Note: this list[^23] of actors is not definite or complete[^24] and I am less confident about this than other parts of the piece.
Xi Jinping. As explained above, Xi Jinping is irreplaceable for both fast and gradual EPC scenarios.
Leaders in the AI tech stack:
a. Leaders of AI labs. The most capable models currently come from Kimi (Moonshot AI), Qwen (Alibaba), and DeepSeek. They are somewhat replaceable because competition between labs[^25] would make it easy to replace them when needed, suggesting that nationalization won’t serve the power seeker(s) because it removes this productive competition.
b. SMIC for accelerating chip manufacturing and achieving chip sovereignty. They are likely irreplaceable because there is no apparent Chinese replacement.
c. Huawei (chip design). Huawei has high leverage in the AI tech stack; apart from their leading Ascend chips, they also produce HBM memory and an AI software stack. They are likely irreplaceable because of the high leverage they have in multiple areas in the AI tech stack.
d. Shanghai Micro Electronics Equipment (Lithography machines): They are irreplaceable. If China developed compute sovereignty, they would probably be the ones developing the lithography machine because they control the expertise and machinery.
MIIT and the Ministry of Finance. They are the key players in the AI big fund and its subsidiary, the National AI Industry Investment Fund, which now funds development in the AI tech stack. They are somewhat hard to replace to accelerate funding.
Top officials at MIIT and CAC (model oversight): They are somewhat replaceable if reports can be fabricated and models learned to game their evaluations. Also, they have limited leverage if there is no concern about other EPC attempts.
Leader(s) of factory workers who build robot infrastructure to automate the economy. Each factory worker will be replaceable, but the thought leaders of the group (either chosen leaders or local governors) would be harder to replace because they may trigger political mobilization.
Members of the security apparatus to monitor actors and enforce commitments by force when needed. They are Irreplaceable, because they protect from human defection within the coalition or resistance from actors outside of the coalition.
My take is that the likelihood of this Fast EPC scenario is contingent on Chip sovereignty in China and the political skills of power seeker(s) to create an effective power-sharing agreement with other actors controlling EPC conditions.
5. Barriers for EPC
There are several barriers for EPC; some of them are mentioned throughout the piece. Below, I flesh them out here again, starting with the cross-cutting barriers before explaining the barriers for each scenario.
5.1 Cross-cutting Barriers
Lack of interest: It’s possible that actors won’t like to live in the EPC world we described, so they don’t opt for EPC or instead use their EPC power for good. This also applies to AGI. It’s not clear that Xi or CCP officials are AGI-pilled. It is true that the CCP is now diffusing AI to realize economic growth, but they don’t seem very excited about building AGI that surpasses human capability. However, this is hard to verify, and therefore we can’t lean on it. Also, this barrier is more relevant to Fast EPC than Gradual EPC, because for the latter it can happen without actors intending it.
The US wins the race: China is currently lagging behind the US frontier by several months. Therefore, the US winning the race would probably block Chinese power seeker(s) from realizing EPC. This is because the first mover would have an advantage in shaping which post-AGI world we end up in.
Technical barrier. We may fail to build AGI. Also, we may fail to align AI with human preferences, and the risk of misaligned AI would stand in the way of EPC attempts.
Xi’s mortality: If Xi dies, it would be hard for his successor to maintain the one-man rule, and CCP members may engage in defensive political mobilization to call for collective leadership in the CCP, which would make it hard to get centralized leverage over the institutional conditions.
5.2 Barriers for Gradual EPC
The main technical bottleneck is the ability to verify alignment and loyalty using white-box (non-behavioral) techniques. Unlike censorship, where a keyword list would signal misaligned content, the feedback of EPC conditions (e.g., a model with secret loyalties or misaligned AI) is harder to get because the model may game the behavioral evaluation. That said, as the field of AI evals grows, we may find answers to these open questions.
5.3 Barriers for Fast EPC
Political barrier: CCP members who know about the EPC plan but have not been included in the power-sharing agreement may try to sabotage the EPC attempt.
Power conflict: Power seekers would end up sharing AI power and expanding the leverage of other actors within the coalition to incentivize them, which in turn makes it a non-extreme concentration of power because a conflict within the members of the coalition may decentralize power. That said, this is very speculative, and this power conflict is not necessarily beneficial for those outside of the coalition.
Public resistance: It seems to me one of the main reasons for CCP legitimacy (at least in their narrative) is its promise of prosperity and stability. Currently, the public has some indirect leverage over the Chinese state because they work[^26] and finance innovation through their cheap savings, so their contribution is vital for realizing economic and technological growth. This is why “social stability” is a metric CCP members optimize for, and it’s what gives the public the indirect (‘relatively small’) leverage they have over the CCP. This will not continue to be the case under the EPC scenario, so you can imagine that if people suddenly lost their jobs to AI without sustained guarantees that they would be given UBI, they would be incentivized to collectively organize to engage in sabotage of AI infrastructure or kill members of the EPC coalition.
As a non-Chinese person, I put more weight on public resistance than the Chinese and China experts I spoke with. They think CCP legitimacy is more complicated than mere economic legitimacy, drawing on the fact that increasing unemployment rates and the “COVID Zero policy” incidents did not trigger collective action. So it seems like the Chinese public is too risk-averse to engage in such action. This is triggered by the strong surveillance machine that would stand in the way of any collective organizing and the history of failed attempts at public organizing. That said, given the large scale of EPC risks, I still think that, absent countermeasures from the CCP to address job displacement, the public will see enough signs that would help them collectively internalize disempowerment, and the AGI world would be seen as riskier than defection.
6. So what can we do about Fast and Gradual EPC in China?
My take is that while EPC scenarios have limited likelihood in China, I think they are worth worrying about given the severe consequences if we did not prepare for them. Also, I think less extreme forms of these scenarios are also worth worrying about because:
It would be worse than the current state of society (think of the current freedom level in China as the best case we can experience in that world).
Actors without control of EPC conditions won’t have leverage and will become a permanent underclass.
Value is locked in forever.
My recommendations on potential areas for interventions are:
The most concrete/doable intervention: Continue to apply pressure on the US to win the AI race. This can happen through various means, including advocacy, lobbying, and engagement with think tanks that already have an established pathway to influence US-China relations (e.g., CSET, RAND). Also, while export control measures have worked to limit China’s access to lithography and high-bandwidth memory, it’s important to address the smuggling networks in Southeast Asian hubs (e.g. Malaysia and Singapore) by supporting existing export measures in these countries.
That said, this “US should win the race” intervention may backfire if it triggers escalation from China’s side, as it would make the race dynamics more intense, something I have already cited as a trigger for the Fast EPC scenario. My guess is that it’s worth the risk only if we are confident that the lag between the US and China is hard to bridge. This means if there is any chance for China to win the race through sudden innovations, or if we realize that China is strategically hiding/misrepresenting information about its most capable models, then this intervention should be replaced with bilateral agreements on slowing down the AI race between the US and China.
As a citizen of the global majority, I am not confident about the quality of the world we would end up with if the US won the AI race, and even for US citizens, it seems like the possibilities for power concentration in the US are very concerning, so I am not saying that EPC in the US is better than EPC in China, but what I am saying is that we have a better chance of stopping EPC in the US than in China if we started working on that now (before AGI), given the democratic nature of the US institutions relative to the autocratic institutions in China.
The hardest and most speculative intervention: raising awareness. The rationale for this intervention is that if actors realized this, they would start to bargain for their contribution in exchange for power. This includes junior CCP members, the developers, factory workers, the public, and any foreign actors that contribute to the supply chain of AI in China. If enough actors realized the possible EPC future and worked to maximize their leverage, we would end up in a world where power is more decentralized than if we did not.
Talk with international organizations and news outlets (e.g., Human Rights Watch, Freedom House, or The Economist) that work with Chinese journalists and lawyers to get information about EPC inside China. The aim is that they would find a way to talk with Chinese people while they are overseas or through the channels they built to smuggle information (through coded communication, steganography, VPN, or through channels like China Digital Times that work to address censorship).
Talk with those working near China. It may be easy to influence China from places geographically near China because they are more aware of the context and may have access to Chinese networks; my guess is that actors in Neo Research and AI safety hubs in Hong Kong, Taiwan, and Singapore may have Chinese networks and practical tips and tricks for raising awareness inside China.
Support AI安全开放社区 | Open Community for AI Safety China, which is the Chinese equivalent of BlueDot Impact, working on educating the Chinese public about AI safety more broadly. These AI safety field-building interventions may prove to be useful because they can help build the community infrastructure and trigger the paranoia toward AI acceleration without talking directly about EPC risk.
One important thing to point out is that, in terms of influencing EPC trajectories from within China, the trust these actors have (something I, as an outsider, don’t possess) is very crucial for communicating this because it’s easy for people to reject speculative scenarios and associate them with conspiracy theories. Even if we have concrete evidence for fast EPC in China (which we don’t, because China is a black box and generally because it’s strategic to keep a low profile about efforts for power concentration), some actors would be under the influence of the propaganda efforts and would think Xi would never want this. The China insiders would know how to communicate this to CCP officials, AI businessmen/businesswomen, developers, academics, and the public. Apart from having networks, these China experts possess important skills that make them more likely to be heard by Chinese actors:
They speak Mandarin
They cite Chinese sources[^27]
They are also very strategic about their framing; for example, instead of saying we should be worried about EPC in China, they say, “How should we make sense of the recent American concerns about EPC in their executive branch?”
Practise strategic framing for the concrete interventions needed to mitigate EPC. For example, if we found that measures like monitoring, maintaining human-in-the-loop, or separation of duties would protect from EPC, we should strategically package them for actors in the technical AI safety space in China (e.g. Shanghai AI Lab, Tsinghua University, etc.). The trick here is that the framing of these interventions should use AI misuse and/or AI loss of control risks rather than EPC because these alternative framings have gained attention in China, so there is already political will to address them. This also suggests prioritizing cross-cutting interventions (ones that address EPC, misuse, and loss of control) over EPC-specific interventions.
We need more people to think about the possibility of extreme power concentration in China. While in this piece I conclude that it’s harder to trigger than what may seem from the outset, I am far from confident about this conclusion because the strategic ambiguity of working towards EPC makes it hard to trace signals for its likelihood. More CCP EPC-related projects in the AI safety fellowships (e.g. IAPS, GovAI, MATS, SPAR, etc.) and AI safety RFPs may help us find the right pool of talent to continue thinking about this.
I am extremely grateful to my mentor and the many anonymous reviewers for their constructive feedback that helped me refine my hypothesis.
[^1]: There are two reasons for this:
> distillation will not be enough to outperform the base model capabilities.
> current [distilled models (e.g. DeepSeek) have turned out to be politically aligned with Western political views](https://jebasania.substack.com/p/a-shallow-snorkel-into-the-political).
[^2]: While power concentration is also concerning, and we should spend time thinking about what we can do about it, the scope of this project was only on extreme power concentration. Future pieces can further investigate power concentration in CCP, one that enforces CCP surveillance, censorship, and loyalty screening without making power concentrated within an extremely small group.
[^3]: Unless the US respects China’s sovereignty after reaching AGI which I think is extremely unlikely.
[^4]: I am very concerned about extreme power concentration risk in the US but the reason why I am calling for this is because I think we have a higher chance of addressing the risk in the US than China due to the autocratic nature of institutions in China.
[^5]: Before the industrial explosion, they would have been advised by AIs, and they were useful because of their ability to manage human relationships. I argue that after the intelligence explosion a decision would be made to replace them completely with AI because the human officials would be slowing down the progress.
[^6]: Before AI, working at CCP was considered to be the best in terms of job security, social benefits, and social status.
[^7]: It is possible that these programs have been incentivized by the need for human work in the first place so the specification of the post-AGI jobs would be different.
[^8]: People in China already study Xi Jinping’s thought, but what I am imagining is similar to the Cultural Revolution era, where it would be the sole subject people studied.
[^9]: These jobs seem to benefit from human worker more than AGI robots because of their relational aspect.
[^10]: This is inspired by what happened to the uyghurs in the detention camp, where they have listened to party propaganda for hours.
[^11]: I have thought less about the implication for the dynamics between Xi and the AIs on the remainder of the piece.
[^12]: Unless there was a disagreement inside the elite group or if the AI turned out to be misaligned.
[^13]: Some of the people I spoke with find this hard to believe and some of them think non biological means like brain uploading are more likely than biological immortality. I have chosen biological immortality because it seems like the more desirable goal for the power seeker(s) even though it seems more unlikely at the moment.
[^14]: The tight party control also influences what subordinates decide to report, as it may incentivize actors to be selective with the bad news they share because they fear for themselves. This was seen clearly in Mao’s Great Leap Forward, where he set unrealistic goals, and his subordinates hid early warnings about the famine.
[^15]: I was inspired by Nick Cheesman’s analysis of the reforms in Africa, where he cites the death of leaders as the major contributor.
[^16]: The term limit and the succession dynamics were one of the great checks on power in China (a country with no elections and a single party). Each party chief was expected to serve only for 2 terms (10 years) to avoid actors controlling power forever.
[^17]: He is now 73 years old, and we can’t predict his death, especially because Xi hides information about his health.
[^19]: Some of the people I spoke with think this is not necessary. I disagree for three main reasons:
> AI surpassing human intelligence would remove the cultural barrier to human replacement.
> We would still have intelligent humans on some posts, and these humans would have leverage and therefore constrain the agency of the power seeker(s).
> Also, even if there were very strong incentives for EPC before AGI, I think it would probably take time to get enough AI diffusion and institutional reforms even if AGI was not a requirement, so it’s possible that if we continued to advance AI, we would get AGI by then.
[^20]: It’s important to clarify that some AI developers in China are more sovereign than others; for example, Alibaba has its own compute infrastructure.
[^21]: I am personally not excited about delegating AI evaluation to AI judges because of validity and human disempowerment concerns.
[^22]: Social media companies hire censors to ensure compliance with party requirements.
[^23]: I used Claude to help me map stakeholders in the Chinese AI tech stack
[^24]: For example, other actors may include actors from propaganda departments to obscure signals about EPC. Also actors in control of minerals and energy sources.
[^25]: Also, the open-weight models limit the leverage each individual lab has.
[^26]: Because simply they can passively resist doing the work or do it terribly.
[^27]: The reliance on Chinese sources and knowledge turns out to be a big deal for Chinese and there is distrust toward those who over-rely on western sources.
You describe a scenario in which biological immortality and a post-work society have both been achieved. Studying Xi Jinping Thought and spending a few years doing makework seem a small price to pay for access to this! The part about being tortured with an experience machine seems an arbitrary addition. Your scenario already includes superpersuasion, isn't that enough to change the minds of any dissenters? To say nothing of truly advanced technology that can just modify brains directly.
So the description of your scenario does not inspire me to resist it. It contains features (immortality, universal leisure) that we will be extremely fortunate to obtain under any transhuman regime, and also a negative feature (torture via neurotechnology) which does not seem inherent to the scenario (EPC under CPC).
On Positionality: I am a non-Chinese person; I don’t speak Mandarin, and I have not lived in China before. I have written this piece after reviewing English publications on Chinese politics and AI-enabled extreme power concentration risk for ~12 weeks. I tried to account for my detachment by requesting feedback from people with China expertise.
Summary
I define Extreme Power Concentration (EPC) here as economic and coercive power being concentrated in the hands of the few who control the most powerful AI. The emphasis on the few is very important because the number of actors who end up in power differentiates between a world where power is “extremely concentrated” and another where power is "concentrated." Despite the recent attention to the AI-enabled extreme power concentration problem, little work has investigated what would happen if China-the first runner-up in the AI race happens to develop AGI first. The disparity between the American political system and the Chinese political system further limits the transferability of the current threat models and interventions to China and thus makes us less prepared for the manifestation of CCP-led EPC risk.
Through this project, I tried to assess whether the theoretical argument of AGI enabling ~ one person or an extremely small group to independently run an entire country (in this case, China) is plausible given the seemingly high propensity for one-man rule in China, and if so, what can we do about it. I start from the assumption that China managed somehow to secure the lead in the AI race over the US and that it solved its compute bottleneck and achieved chip sovereignty. Therefore, model distillation pathways won’t be sufficient for this scenario[^1].
My thesis is that while AI would make power more concentrated[^2] in China, extreme power concentration (EPC) is hard to materialize in the CCP. I argue that in a Fast EPC scenario, the power seeker would need buy-in from other actors in the AI tech stack and CCP officials to make it happen. Therefore, incentivising those actors (to create a world in which they won’t be needed anymore) would break the main assumption EPC relies on (that power seeker(s) can do things without other humans if they wanted to), given that these other actors realize the power seeker(s) plan and demand sustained leverage. This makes it hard for the power seeker to realize EPC with a very small number of actors. That said, it can be possible to realize Fast EPC with a relatively large coalition of actors governed by a power-sharing agreement.
In a gradual EPC scenario, cooperation for EPC would be a survival strategy and effectively the norm, and other actors may not realize the world they are heading towards; therefore, their power would decrease gradually. However, given the US lead in the AI race dynamics between the US and China, we are more likely to live in a Fast EPC scenario than a Gradual EPC scenario because I assume that China must build AGI first to realize EPC[^3]. Also, EPC is more likely under Xi’s term than under his successors due to the personal loyalty Xi has managed to build over the years. This loyalty would make it easier for him to realize extreme power concentration conditions.
The main conclusion of the piece is that we should talk more about EPC to help both Chinese actors and US policymakers internalize it and develop a healthy paranoia toward AI acceleration in China. We should think about potential interventions for this risk despite its low probability because some of the interventions could benefit less extreme scenarios of the risk which are more likely. Particularly, I recommend:
The piece starts with a detailed EPC scenario to help readers internalize what this EPC world may look like before discussing how EPC would change dynamics within the CCP, conditions for realizing EPC, potential models for EPC, barriers to EPC, and interventions for EPC.
1. What does this EPC world look like?
Note: this scenario is an attempt to help the readers picture the worst-case version of the extreme power concentration risk under CCP. It’s not a forecast, so if some of the assumptions here seem implausible, feel free to skip ahead; the arguments in the rest of the piece do not depend on the speculative details here.
Imagine this: we are in 2035, and Xi Jinping is still alive. In the last ten years, China launched its AI++ plan, made unexpected progress in its chip industry, and managed to reach Artificial General Intelligence (AGI), which surpasses human capacity in all tasks. The AGI robots can do anything autonomously. They can draft CCP governance plans for the 1.41 billion people, implement them at the local level, and report the progress to Xi. This is not specific to economic functions but also to national security functions. AI has been integrated into the People's Liberation Army (PLA) with fully autonomous weapons and AI strategic planners that made an astonishing victory in Taiwan, and the rest of the world, including the US, either surrendered or was defeated and became part of the greatest China.
1.1 What does this world look like for CCP members?
This means that for most of the 101 million party members, AGI has taken their job[^5]. The party members did not expect this because they were optimizing for AGI development, and there was some sort of pact that they would maintain their jobs after integrating AI advisors[^6]. It was difficult to predict this because conversations on AI risks were censored, and party propaganda made it seem like the CCP has control of AI risks and that AGI will serve socialism with Chinese characteristics and the great rejuvenation of the Chinese nation.
1.2 What does this world look like for Chinese citizens?
Xi continued to oppose welfarism. He thinks people should not be granted wealth without working for it. So he invented jobs (similar to inmate work programs[^7]) for humans as a way of disciplining themselves and also as a condition to qualify for universal basic income (UBI). These jobs include studying and writing extensively on Xi Jinping thought[^8]. Other jobs include teaching, art production, and nursing[^9]. If you started thinking this future is not too bad, just consider the fact that you won’t get a chance to choose what your job is, and you basically can’t quit. There is no attractive future trajectory that you can compete for (one in which you would get a better quality of life if you have worked hard for it). Also, ordinary citizens do not have any guarantees that Xi won’t wake up one day and decide to cut off all the UBI, apart from good faith in Xi’s moral compass. In this world, all people get the UBI amount determined by the AGI. As far as we know, the criteria it considers to determine UBI are whether a person:
The CCP’s definition of an ideal Chinese citizen (beliefs, social attitude, etc.) will be perfectly enforced. If you feel like this is already happening in China, imagine the enforcement power AGI can bring. Even for choices like marriage, people would have to do it the CCP way; the AGI matching algorithm would pick their potential human/AI partner and later the eligibility for having kids. The AGI would be the CCP’s “angel on the shoulder”, guiding you in every social situation to behave in a way that Xi endorses.
All people will be made atheists by talking for hours to a super-persuasive AI that is trained on party propaganda, and there would be no need to manage minorities like Uyghurs, or Tibetans because everyone is part of the majority engineered by the CCP. There is no other place to escape to because Xi has power over all countries and even space.
You can’t pretend in this world because the surveillance is stronger than ever; using brain signals, Xi can get access to people's ideas and feelings even if they did not express them.
For those who resist, they get into a brain simulation equivalent to an “experience machine,” but one that answers for Xi only, and it optimizes for suffering and pain for his defectors. In this machine, you would feel all the horrible things a person can feel during multi-round simulations (death of your loved ones, painful disease, starvation, physical and sexual violence). Compared to the CCP experience machine, the horrible suffering currently happening at the CCP detention camp would be considered merciful.
1.3 What does this world look like for non-Chinese citizens?
For those who are non-Chinese, the situation is even more difficult. Under the unification vision, Xi Jinping is turning the whole world into the greatest China. Foreigners are expected to immerse themselves in Chinese culture and values through talking for fixed hours with superpersuasive AIs[^10].
After being cleared from the loyalty screening, foreigners are eligible for half of what UBI recipients in mainland China get, and as an act of revenge, anyone who ever shared negative views about the CCP before would experience the suffering machine (this piece would put me in a disadvantaged position in such a world).
1.4 What does this world look like for AI?[^11]
Xi transferred most of his propaganda techniques to the AI alignment process, which resulted in AIs that take Xi’s thought as ground truth and are internally motivated to advance Xi’s interest. While he resisted AI welfare policies at the beginning, he slowly extended his views on welfarism to AI, meaning that welfare won’t be granted to AIs unless the AI proves its worth and loyalty to Xi.
1.5 On the possibility of transition
The main issue is that there is no way for anyone to change this state of society. Value is locked in, and whatever Xi sees as best will continue to determine what’s best for everyone forever.[^12] Xi never dies in this world because his new AGI-led health preservation center discovered effective drugs for immortality, so his term effectively becomes unlimited[^13].
2. How AGI May Change Power Dynamics Within CCP?
Before discussing the likelihood of this EPC world, I want to make the case that if it happened, it would change the power dynamics not only between the CCP and the public but also within the CCP.
My initial question about AI-enabled power concentration in China is, "Isn't power already concentrated in China?" The answer is yes: power in China is already concentrated, but AGI will make this worse. Contrary to my initial assumptions, there are dynamics inside the CCP that decentralize power “to some extent,” and conditional on the likelihood of EPC, AGI could remove them. Below, I try to analyze the current decentralization dynamics and how AI may change them.
2.1 Delegation
The practical limitation of governing 1.41 billion people and the need to delegate governance tasks to regional and provincial secretaries decentralize power in China. The need to incentivize these delegates limits the power of those at the top. This does not mean that other actors engage in active resistance if they are not granted leverage; instead, it’s closer to passive forms of resistance where people do not take initiative or find excuses to defend their inaction.
I argue that AGI will remove the delegation dynamics within the CCP, as AIs replace CCP officials. Below, I list the reasons why Xi would be convinced to replace CCP officials with AGI:
2.2 Competition
The CCP uses competition to incentivize delegates and limit the power of each individual actor, which in turn helps the CCP achieve its governance objectives. The most obvious one is among Chinese businessmen/businesswomen, but the impact of political competition among Xi’s subordinates is often overlooked. The competition between CCP elites over promotion and being Xi’s favorite helped them check each other's power. The competition is a prevalent theme in navigating the job market and the exams in China. So whether the CCP manufactured competition or it happened systematically as a coping mechanism, it tells us that as long as there is a chance for people to “play the game," people are channeling their energy to win by the rules of the game set out by the CCP rather than attempting to change the rules of the game. I argue that, absent countermeasures, AGI may break the social contract for party members and ordinary citizens, where loyal officials who play by the rules of the game won’t get a chance to climb the party ladder or accumulate resources. For citizens, it means that perfect obedience and hard work will not be rewarded (In my scenario above, Xi mitigated this by using UBI as a reward mechanism).
2.3 Information asymmetry
There are logistical barriers to telling superiors everything, and the discrete judgment of deciding what’s important to tell decentralizes power.[^14] AI will remove mechanisms because it will unlock perfect surveillance and centralized processing of information.
2.4 Death[^15]
Although in 2022 Xi Jinping broke the term limit rule[^16], death continues to be an uncontrollable limit for Xi’s power. The impression is that Xi would opt for a fourth term and continue to rule until he dies[^17], similar to Mao, who served as party chief until he died at the age of 82. AGI may change this by unlocking unlimited terms either through new drug discovery breakthroughs in longevity science[^18], or through creating a digital twin of Xi Jinping, an AGI that is fully aligned with Xi’s thoughts.
All of this motivates me to think that AGI would be an attractive asset for power seeker(s), and it would definitely make power concentration worse inside the CCP.
Now that I have established how AI will change the dynamics inside the CCP, I move to explain my best guesses for how we would move to the EPC world through what I call conditions for the EPC before sharing my estimates on its feasibility.
3. Conditions for EPC
There are several technical and non-technical conditions the power seeker(s) will seek to control to achieve their EPC vision. Ideally, the power seeker(s) will attempt to control all of these conditions to eliminate their dependency on others, but realistically, and depending on the speed of the takeoff, they would end up in a world where they get buy-in from others, who in turn control the conditions via a power-sharing agreement. Below, I classify them into technical, institutional, and technical governance conditions.
3.1 Technical Conditions
These conditions are mainly controlled by people with technical expertise or those who have control over them. These technical experts may include developers in the AI labs and academics at leading universities in AI research (e.g., Tsinghua University, Peking University, etc.).
3.2 Institutional Conditions for EPC
Direction: What Gets Built
Resources
Controls after deployment
3.3 Technical Governance Condition
That said, this evaluation knowledge (without other conditions) won’t be sufficient for EPC. Also, there are two possibilities under which the bargaining power of these evaluators may become smaller if:
3.4 Why is EPC led by leaders at AI labs alone not feasible?
The direction, verification, and deployment conditions are where Xi and party members would have the most leverage over actors in the private sector. This is because, although in practice private actors have leverage on development and they can build models that are secretly loyal to them, they would struggle to arrange this under the CCP surveillance machine. And even if they coordinated that outside China, they would need actors in the CCP to cooperate with them by:
And while, in theory, these businessmen/businesswomen can use the race to legitimize cutting corners to get buy-in from some actors, there is no guarantee that CCP officials would help them at all times over their competitors; also, the lack of incentives for these CCP officials would stand in the way of getting things done quickly and efficiently, a prerequisite for Fast EPC. This motivates me to think businessmen/businesswomen led EPC in China is hard to achieve.
3.5 Why is EPC Without Xi not as likely?
I argue that if Xi dies, his successor would find it harder to concentrate power via AI because it takes time to build personal loyalty and tune the institutions in a way that serves the power seeker's interests (consider how Xi in his first term was relatively weaker than his current version in the third term). Moreover, it seems like power concentration in the CCP is not only about the post but also about actors’ agency and political skills, which influence the space of possibilities. For example, Xi Jinping was able to concentrate power (via removal of factions and the second-term limit) in a way his predecessor (Hu Jintao) could not. My take is that depending on the speed of the scenario, if we end up in a CCP EPC world, it will probably be led by Xi alone or Xi along with other actors in the AI tech stack, as I will explain in the upcoming section.
4. Modeling Potential Scenarios
I consider two scenarios for EPC. Below, I explain their implementation, prerequisites, consequences, arguments against their likelihood, and tentative signposts to watch for.
Scenario 1: Gradual EPC
Scenario 2: Fast EPC
Speed of the takeoff
Gradual and relatively slow
Fast
Actor(s) that end up with power
Xi Jinping alone
Coalition of actors
Other actors’ leverage
Low because their cooperation gradually becomes the norm.
High because their fast cooperation is needed.
What is needed for this scenario?
Diplomacy or technical innovation to block the US from reaching AGI first. Strengthen evaluation capabilities for secret loyalties and evals gaming.
China builds chip sovereignty. Incentives for actors to accelerate development.
Arguments against its likelihood
Stalling the US from building AGI is unlikely to succeed for long and the US is more likely to build AGI first.
Fast EPC is hard because a large number of actors may not cooperate without sustained leverage. It may be technically infeasible to build chip sovereignty fast
What to watch for: tentative signposts
No major changes in CCP dynamics, and Xi is willing to take a fourth term. China tightens AI evaluations. The AI dialogues between the US and China succeeded in pacing the frontier temporarily
The increased Chinese investment in chip sovereignty. AI acceleration continues.
The distinction is important because actors’ leverage, power distribution, and therefore entry points for interventions would be different under each scenario. Below, I try to explain each scenario in more detail.
4.1 Gradual EPC Scenario: Default Pathway
Models are already being evaluated for behavioral alignment with CCP values (which include Xi’s thoughts). So unless the US built AGI first, I argue that we will end up in a gradual EPC world whether Xi intends this or not. This can happen by:
All the conditions I discussed above are somewhat controlled by the CCP, so apart from creating strict AI evaluation to tighten party control, nothing has to change to realize gradual EPC. The way I imagine this to happen is as follows: extreme party control and the competition between labs would create a race-to-the-top dynamic in which AI labs optimize for alignment with Xi, slowly moving towards AI henchmen without realizing it. It can be strategic or systemic. Similar in speed to gradual disempowerment scenarios.
In the gradual EPC, the compute and robotics infrastructure would be gradually built by humans at the same time we are accelerating AI development, and when we reach software-only AGI, we would have most of the infrastructure needed to automate the remaining parts, so there would be no single point in the future at which we would start building the infrastructure for industrial explosion and military AI.
4.1.1 Why Would AI Companies Cooperate in the Gradual Scenario?
The initial question I had is, why would AI companies and other actors do that? Why wouldn't they bargain to sustain their leverage in this gradual EPC world?
The answer to this can be inferred from the censorship analogy. Aligning AI to Xi’s thoughts and perfect obedience to Xi can be implemented in a way that is similar to censorship, where the social media companies just do what the party wants, or else they would be shut down. I argue that the key aspect for strong enforcement is the competition and the delegation of censorship to the companies[^22]. Beyond AI-driven capacity gains, time has also significantly strengthened the censorship machine, as practices evolved from simple to sophisticated approaches that are hard for citizens to game, and, most importantly, time helped the public and the social media companies normalize censorship. This is the exact mechanism I am imagining to see with AI alignment and evaluations, where Chinese institutional genes will pave the path towards AI-enabled extreme power concentration; this will happen gradually by triggering a race to the top between Chinese labs to ensure their models are fully aligned with Xi. Current evaluation dynamics already replicate the censorship dynamics where the responsibility of running evaluation (e.g. filing for CAC review and ethics boards) is mostly delegated to the AI labs.
The leverage other actors would have in this gradual EPC scenario is limited because, as in censorship, the gradual nature makes it easy to normalize it. Also, the incentive structure will reflect this and reward cooperation in the short term. People would not internalize the risks because they would be so busy with the short-term incentives of cooperation (getting promotions and funds for accelerating EPC). Cooperating with Xi to realize EPC would be the new normal stable state of Chinese society; therefore, collective action (among actors in the private sector, CCP officials, or the public) to change this would become hard to trigger, and Xi would have enough time to strategize about EPC (for example, introduce UBI before Chinese people fully internalize the risk of job displacement).
Interestingly, the task of political evaluation of public-facing AI models initially was similar to censorship with a keyword list to control what the model outputs to the public, but now, as the shift has transitioned from what the AI says to what the AI does and values internally, the task of evaluation with Xi’s values to achieve Gradual EPC is likely to be bottlenecked by technical gaps rather than political will within China, as we will explore in the barriers section.
4.1.2 Gradual EPC and the US lead
The main blocker for Gradual EPC is whether the US built AGI first, because it’s not clear that if the US reached AGI first, it would not interfere with China. After all, they would be too powerful, and there is no global government to check their power. To that extent, China's current efforts to call for global AI governance may be seen as a defensive move to ensure nations' sovereignty in the post-AGI world. This can enable Gradual EPC in China even if the US reached AGI first, but I am not confident that this is feasible. An easier strategy for Chinese power seeker(s) may be to use current Trump-Xi dialogue momentum to convince the US to slow down because of false positive loss-of-control risks in the AI notification mechanism while China secretly works to bridge the gap.
4.2 Fast EPC
While gradual EPC is the default pathway, its likelihood is undermined by the possibility of the US winning the race first. Therefore, under AI race dynamics, the power seeker(s) may attempt to speed up EPC.
The way I imagine this to happen is that Xi would incentivize actors in the AI tech stack to cooperate and build AGI fast to achieve the great rejuvenation of the Chinese nation. Similar in speed to AI-enabled coup scenario. After achieving chip sovereignty and reaching software-only AGI, human workers would be tasked with the physical labour of building the compute and the robotics infrastructure to facilitate full automation of the economy and military AI.
I argue that the most likely scenario for a fast EPC is a coalition of actors rather than a single actor because actors would require incentives to work toward realizing Fast EPC. And while negative incentives can be easier to introduce, they result in more risk-averse behaviour, the opposite attitude for AI acceleration and development. Also, unlike a gradual scenario, the power seeker(s) don’t have the privilege of time to strategize and tune institutions gradually; therefore, other actors (developers, factory workers, etc.) would have more leverage than in the gradual EPC scenario. To secure buy-in, the power seeker(s) will probably start with political mobilization and manipulation using the Chinese rejuvenation slogans to incentivize actors first before proposing a share in the AI power because It’s in the power seeker(s)’ best interest to try to minimize the leverage other actors have in the EPC world. Once this strategy fails with some actors, depending on their leverage, the power seeker(s) may propose a power-sharing agreement and offer them:
Also, for each condition, only actors at higher layers (top officials, CEOs) would likely be included in the coalition, whereas lower layers (AI engineers, builders of infrastructure, junior CCP officials) would not be given high leverage unless their manipulation by their leader is not possible and they are irreplaceable.
Below, I try to list potential actors in the Fast EPC coalition with my guesses on how replaceable they would be for the Fast EPC scenario.
Note: this list[^23] of actors is not definite or complete[^24] and I am less confident about this than other parts of the piece.
My take is that the likelihood of this Fast EPC scenario is contingent on Chip sovereignty in China and the political skills of power seeker(s) to create an effective power-sharing agreement with other actors controlling EPC conditions.
5. Barriers for EPC
There are several barriers for EPC; some of them are mentioned throughout the piece. Below, I flesh them out here again, starting with the cross-cutting barriers before explaining the barriers for each scenario.
5.1 Cross-cutting Barriers
5.2 Barriers for Gradual EPC
5.3 Barriers for Fast EPC
As a non-Chinese person, I put more weight on public resistance than the Chinese and China experts I spoke with. They think CCP legitimacy is more complicated than mere economic legitimacy, drawing on the fact that increasing unemployment rates and the “COVID Zero policy” incidents did not trigger collective action. So it seems like the Chinese public is too risk-averse to engage in such action. This is triggered by the strong surveillance machine that would stand in the way of any collective organizing and the history of failed attempts at public organizing. That said, given the large scale of EPC risks, I still think that, absent countermeasures from the CCP to address job displacement, the public will see enough signs that would help them collectively internalize disempowerment, and the AGI world would be seen as riskier than defection.
6. So what can we do about Fast and Gradual EPC in China?
My take is that while EPC scenarios have limited likelihood in China, I think they are worth worrying about given the severe consequences if we did not prepare for them. Also, I think less extreme forms of these scenarios are also worth worrying about because:
My recommendations on potential areas for interventions are:
That said, this “US should win the race” intervention may backfire if it triggers escalation from China’s side, as it would make the race dynamics more intense, something I have already cited as a trigger for the Fast EPC scenario. My guess is that it’s worth the risk only if we are confident that the lag between the US and China is hard to bridge. This means if there is any chance for China to win the race through sudden innovations, or if we realize that China is strategically hiding/misrepresenting information about its most capable models, then this intervention should be replaced with bilateral agreements on slowing down the AI race between the US and China.
As a citizen of the global majority, I am not confident about the quality of the world we would end up with if the US won the AI race, and even for US citizens, it seems like the possibilities for power concentration in the US are very concerning, so I am not saying that EPC in the US is better than EPC in China, but what I am saying is that we have a better chance of stopping EPC in the US than in China if we started working on that now (before AGI), given the democratic nature of the US institutions relative to the autocratic institutions in China.
Raising awareness sounds reasonable, but if you have read about the censorship in China, you probably know that this is a dead end. Anything that would trigger collective action would be censored regardless of the topic. However, there are many concrete paths we could take:
One important thing to point out is that, in terms of influencing EPC trajectories from within China, the trust these actors have (something I, as an outsider, don’t possess) is very crucial for communicating this because it’s easy for people to reject speculative scenarios and associate them with conspiracy theories. Even if we have concrete evidence for fast EPC in China (which we don’t, because China is a black box and generally because it’s strategic to keep a low profile about efforts for power concentration), some actors would be under the influence of the propaganda efforts and would think Xi would never want this. The China insiders would know how to communicate this to CCP officials, AI businessmen/businesswomen, developers, academics, and the public. Apart from having networks, these China experts possess important skills that make them more likely to be heard by Chinese actors:
I am extremely grateful to my mentor and the many anonymous reviewers for their constructive feedback that helped me refine my hypothesis.
[^1]: There are two reasons for this:
> distillation will not be enough to outperform the base model capabilities.
> current [distilled models (e.g. DeepSeek) have turned out to be politically aligned with Western political views](https://jebasania.substack.com/p/a-shallow-snorkel-into-the-political).
[^2]: While power concentration is also concerning, and we should spend time thinking about what we can do about it, the scope of this project was only on extreme power concentration. Future pieces can further investigate power concentration in CCP, one that enforces CCP surveillance, censorship, and loyalty screening without making power concentrated within an extremely small group.
[^3]: Unless the US respects China’s sovereignty after reaching AGI which I think is extremely unlikely.
[^4]: I am very concerned about extreme power concentration risk in the US but the reason why I am calling for this is because I think we have a higher chance of addressing the risk in the US than China due to the autocratic nature of institutions in China.
[^5]: Before the industrial explosion, they would have been advised by AIs, and they were useful because of their ability to manage human relationships. I argue that after the intelligence explosion a decision would be made to replace them completely with AI because the human officials would be slowing down the progress.
[^6]: Before AI, working at CCP was considered to be the best in terms of job security, social benefits, and social status.
[^7]: It is possible that these programs have been incentivized by the need for human work in the first place so the specification of the post-AGI jobs would be different.
[^8]: People in China already study Xi Jinping’s thought, but what I am imagining is similar to the Cultural Revolution era, where it would be the sole subject people studied.
[^9]: These jobs seem to benefit from human worker more than AGI robots because of their relational aspect.
[^10]: This is inspired by what happened to the uyghurs in the detention camp, where they have listened to party propaganda for hours.
[^11]: I have thought less about the implication for the dynamics between Xi and the AIs on the remainder of the piece.
[^12]: Unless there was a disagreement inside the elite group or if the AI turned out to be misaligned.
[^13]: Some of the people I spoke with find this hard to believe and some of them think non biological means like brain uploading are more likely than biological immortality. I have chosen biological immortality because it seems like the more desirable goal for the power seeker(s) even though it seems more unlikely at the moment.
[^14]: The tight party control also influences what subordinates decide to report, as it may incentivize actors to be selective with the bad news they share because they fear for themselves. This was seen clearly in Mao’s Great Leap Forward, where he set unrealistic goals, and his subordinates hid early warnings about the famine.
[^15]: I was inspired by Nick Cheesman’s analysis of the reforms in Africa, where he cites the death of leaders as the major contributor.
[^16]: The term limit and the succession dynamics were one of the great checks on power in China (a country with no elections and a single party). Each party chief was expected to serve only for 2 terms (10 years) to avoid actors controlling power forever.
[^17]: He is now 73 years old, and we can’t predict his death, especially because Xi hides information about his health.
[^18]: I am imagining something similar to Russia’s state led longevity center.
[^19]: Some of the people I spoke with think this is not necessary. I disagree for three main reasons:
> AI surpassing human intelligence would remove the cultural barrier to human replacement.
> We would still have intelligent humans on some posts, and these humans would have leverage and therefore constrain the agency of the power seeker(s).
> Also, even if there were very strong incentives for EPC before AGI, I think it would probably take time to get enough AI diffusion and institutional reforms even if AGI was not a requirement, so it’s possible that if we continued to advance AI, we would get AGI by then.
[^20]: It’s important to clarify that some AI developers in China are more sovereign than others; for example, Alibaba has its own compute infrastructure.
[^21]: I am personally not excited about delegating AI evaluation to AI judges because of validity and human disempowerment concerns.
[^22]: Social media companies hire censors to ensure compliance with party requirements.
[^23]: I used Claude to help me map stakeholders in the Chinese AI tech stack
[^24]: For example, other actors may include actors from propaganda departments to obscure signals about EPC. Also actors in control of minerals and energy sources.
[^25]: Also, the open-weight models limit the leverage each individual lab has.
[^26]: Because simply they can passively resist doing the work or do it terribly.
[^27]: The reliance on Chinese sources and knowledge turns out to be a big deal for Chinese and there is distrust toward those who over-rely on western sources.