I've been thinking about this in the back of my mind for a while now. I think it lines up with points Cory Doctorow has made in talks about enshittification.
I'd like to see recommendation algorithms which are user-editable and preferably platform-agnostic, to allow low switching costs. A situation where people can build their own social media platform and install a recommendation algorithm which works for them, pulling in posts from other users across platforms who they follow. I've heard that the fediverse is trying to do something like this, but I've not been able to get engaged with it yet.
It's cool to see efforts like Tournesol, though it's a shame they don't have a mobile extension yet.
1Bluesky has custom feeds that can bring in posts from all platforms that use the AT Protocol, but Bluesky is the only such platform right now. Most feeds I've found so far are simple keyword searches, which work nicely for having communities around certain topics, but I hope to see more sophisticated ones pop up.
I'm a bit surprised this post has so little karma and engagement. I would be really interested to hear from people who think this is a complete distraction.
Good recommendation engines are really important for our epistemic environment, in my opinion more than for example prediction markets. Because it indeed affects so much of the content that people ingest in their daily lives, on a large scale.
The tough question is how tractable it is. Tournesol has some audience, but also seems to struggle to scale it up despite pretty mature software. I really don't know how effective it would be to pressure companies like Facebook or TikTok, or to push for regulation, or to conduct more research on how to improve recommendation algorithms. Seems worth investigating whether there are cost-effective opportunities, whether through grants or job recommendations.
1
1
1"I really don't know how tractable it would be to pressure compagnies" seems weirdly familiar. We already used the same argument for AGI safety, and we know that governance work is much more tractable than expected.
Helping to enforce the DSA might be one opportunity. The DSA is a constraining piece of legislation, but the team tasked with monitoring online platforms and enforce it is understaffed, especially in the early days (these roles were actually on the 80,000 Hours jobs board). So, perhaps there could be an opportunity e.g. in finding ways to help them automatically detect or verify compliance issues, if they accept such contributions.
For Tournesol, their website doesn't appear to have changed so much during the last year, so I suppose it is pretty mature. They also have other projects, and they tend to foster a community of French volunteers interested in recommendation algorithms. It depends on whether such projects could have a large-scale impact.
1I don't Tournesol is really mature currently, especially for non french content, and I'm not sure they try to do governance works, that's mainly a technical projet, which is already cool.
Yup, I think research that studies the effect of recommendation algorithms on the brain, from various social media platforms and compares them to the effects of narcotics, would be extremely useful.
I think we're really really lacking in decent legislation for recommendation algorithms atm - at the absolute bare minimum, platforms which use very addictive algorithms should have some kind of warning label informing users of the possibility of addiction - similarly to cigarettes - so that parents know clearly what might happen to their children.
This is going to be even more important as things like character.ai grow.
1I agree that more research on effect of recommendation algorithms on the brain would be useful.
Also research looking at which cognitive biases and preferences the algorithms are exploiting, and who is most susceptible to these e.g. children, neurodiverse etc. It seems plausible to me that some ai applications e.g. character.ai as you say, will be optimising on some sort of human interaction and exploiting human biases and cognitive patterns will be a big part of this.
What do you think is the main issue preventing companies from making more ethical recommendation algorithms? Is it the difficulty of determining objectively what is accurate and ethical? Or is it more about the incentives, like an unwillingness to sacrifice addictiveness and part of their audience?
I think incentives. Based on my recent reading of 'The Chaos Machine' by Max Fisher, I think it's closely linked to continually increasing engagement driving profit. Addictiveness unfortunately leads to more engagement, which in turn leads to profit. Emotive content (clickbait style, extreme things) also increase engagement.
Tools and moderation processes might be expensive on their own, but I think it's when they start to challenge the underlying business model of 'More engagement = more profit' that the companies are in a more uncomfortable position.
Zvi's latest newsletter has a section on this topic! https://thezvi.substack.com/i/151331494/good-advice
Couple of thoughts:
1. I recently found out about this new-ish social media platform. https://www.heymaven.com/. Good chance they are researching alternative recommendation algorithms.
2. What particular actions do you think rationality/ea community could do that other big efforts have not done enough, e.g. projects by Tristan Harris or Jaron Lanier.
I believe the closest research to this topic is under the heading "Performative Power" (cf, e.g., this arXiv paper). I think "The Age of Surveillance Capitalism: The Fight for a Human Future at the New Frontier of Power" by Shoshana Zuboff is also a pretty good book that seems related.
1
TL;DR: This post is a two-page introduction to risks associated with recommendation AI. The negative externalities of recommendation AI seem neglected, and there might be comparatively effective work at improving governance and deploying better recommendation algorithms that generally keep the world on a more sane track. You might consider this a viable option if your AI timelines are over 10 years.
One aspect not discussed here is why having good recommendation AIs could be incredibly beneficial. Improved recommendation AI could enhance collective epistemic and encourage people to engage with important issues, ultimately advancing AI safety and many other important topics. It would be a key to fostering a flourishing civilization.
Epistemic status: I am still a bit new to this topic, I might be wrong in important ways, but I'm curious what you think about this. I'm interested in constructive feedback and open to revising my ideas.
Thanks for Lê Nguyên Hoang, co-founder of Tournesol, for his comments and contributions to this post.
Introduction
Recommendation AIs are deeply integrated into our daily lives. Although they are often considered valuable tools for personalizing our online experience, they also present risks. These AIs can significantly affect individuals and society as a whole. In 2017, YouTube estimated that, out of the billion hours of video humanity consumed on YouTube, 70% was due to its AI recommendations. Since 2016, there have been more views on YouTube than searches on Google.[1]
Recommendation AIs can be argued to be an existential risk amplifier, by reducing the quality of our information space, weakening democratic institutions, amplifying mistrust and hate, silencing priority topics and facilitating propaganda activities.[2]
Deterioration of Democracy
The proper functioning of democracies relies on access to quality information and quality deliberation. However, recommendation AIs favor certain information at the expense of others, e.g. maximizing engagement often lead to polarization. This might be weakening the quality of democracies worldwide.
Details on the Deterioration of Democracy
Many studies and research have highlighted a correlation between heavy social media use and an increase in political polarization. Recommendation AIs, by prioritizing provocative and emotionally charged content, contribute to creating waves of hate where users are primarily exposed to viewpoints that reinforce their disdain for opposing beliefs.[3]
The Facebook Files revealed that Facebook’s algorithm changes in 2018 favored divisive and controversial content, which in turn would incentivize politicians and other public figures to adopt more extreme positions to maintain visibility and engagement online.[4]. This phenomenon shows how AIs, by amplifying polarizing content, can influence the production of information and its widespread diffusion at the expense of balanced democratic debate.
Numerous reports (v-dem, IDEA) identify the years of massive social media adoption (around 2012) as a phase transition, after which democracies worldwide have declined. This overall threat to self-governance can be regarded as a catastrophic risk for human flourishing. Although direct causality is difficult to establish, the influence of recommendation AIs on this phenomenon deserves serious attention.
Why is this important for AGI safety? I don't want to see the quality of democracy in the US and other countries that are moving towards AGI to deteriorate any further.
Geopolitical and Conflict Risks
Recommendation AIs also have geopolitical implications, amplifying hate speech or promoting narratives that incite violence. They have contributed to exacerbating international tensions and destabilizing entire regions.
Examples of Geopolitical and Conflict Risks
In Myanmar, Facebook's AIs were accused of amplifying hate speech against the Rohingya, a Muslim minority. During the Rohingya genocide, 700,000+ refugees fled abroad, and the number of deaths is estimated between 25,000-43,000. Amnesty International documented that Facebook's systems not only failed to stop the spread of these hateful messages but sometimes promoted them, thus aggravating the situation.[5]
On a global level, the case of the leading democracies, especially the USA, is especially concerning. It includes the rise of QAnon-sympathizing political candidates and the Capitol Riot. Meanwhile, throughout Europe, there have been numerous far-right mobilizations. Finally, in the last few years, large-scale wars have emerged even in more developed regions of the world (Ukraine, Lebanon). The concerns for civil war or World War III have reached a historical level (30% currently on Metaculus) — except that today’s weaponry is far more destructive than it was in 1939.
Why is this important for AGI safety? Because increasing these tensions could increase the chances of an AI race between countries.
Mute News
We can distinguish between "fake news" and "mute news." While false information attracts a lot of media attention, an even deeper problem lies in the lack of visibility of important topics.[6] Current AIs favor divisive and emotionally charged content at the expense of essential subjects.
Examples of Mute News
As an example, the IPCC report was published at the same time as Lionel Messi's transfer to PSG, which resulted in it receiving no visibility. In many cases, the informational crisis is more related to this lack of access to information than to the issue of fake news.
Another example is the ethics and safety of AI systems. While the enthusiasm for ChatGPT’s spectacularity and MidJourney’s images have been widely spread through recommendation AIs, significantly less attention was given to the risks that these systems raised, and even less so to the laws they violate, even when this was acknowledged by their own creators. Similarly, climate change, cybersecurity, recommendation AIs, and AI safety are unlikely to be addressed properly, if the attention they receive (through recommendation AIs and classical media) vanishes.
Recommendation AIs are the main mechanism that could bring these important topics and unknown unknowns to our attention.
Why is this important for AGI safety? Because the fact that most people and policymakers don't know much about AI risks is a huge bottleneck.
Malicious Exploitation of AIs
Recommendation AIs are currently widely exploited by malicious actors. These actors can manipulate AIs to bury certain information under a mass of content or to promote specific narratives, with the aim of manipulating public opinion, spreading propaganda, or destabilizing countries by encouraging certain ideologies.
Examples of Malicious Exploitation of AIs
For example, the French agency Viginum has documented cases of coordinated pro-Russian propaganda networks, as in the "Portal Kombat" report. These networks exploit social media AIs to amplify their message and influence public debate.
Strikingly, Facebook themselves reports that they suppressed 30 billion fake accounts. On many platforms, most accounts should be expected to be fake. These accounts can be used to provide initial boosts to propaganda-aligned content, thereby tricking recommendation AIs into believing that these contents trigger significant amounts of engagements, which may then make such content viral. Another example is the Instagram influencer David Michigan, who is suspected to have purchased millions of fake subscribers to boost his online business. Such attacks are known as poisoning attacks in AI Safety.
Overall, there is a very wide and active disinformation industry online derived from the old SEO optimization industry. Numerous actors exist even in democratic countries, such as Cambridge Analytica (UK), Eliminalia (Spain), Team Jorge (Israel), Avisa Partners (France), and Alp Service (Switzerland). China is suspected of paying 2 million individuals online for their online soft power.
Why is this important for AGI safety? I don't think this point is that important for AGI safety. I might be wrong.
How Could Recommendation AIs Become Beneficial?
Initiatives are underway to ensure that recommendation AIs become tools serving the common good.
In Europe, regulations such as GDPR, the Digital Services Act (DSA), the Digital Markets Act (DMA) and the AI Act lay the foundations for responsible use of these AIs. These regulations aim to protect user data, limit the power of large platforms, and ensure increased transparency in the functioning of these AIs.
The example of Taiwan is particularly inspiring. In 2014, Taiwan initiated a transition to digital democracy, where digital technology is governed by citizens democratically. This approach has created a model where digital tools, including recommendation AIs, are aligned with the values and needs of society. Remarkedly, over the last decade, Taiwan is the only country that has drastically improved democratically, moving from a flawed democracy with little popular trust in the government, to a model that the people want to defend. This is arguably a strong evidence that the transition to digital democracy is both tractable and extremely effective.[7]
Another initiative is that of the Tournesol[8] non-profit, a participatory research project that aims to develop democratic recommendation AIs. Unlike current AIs that are optimized to maximize engagement, Tournesol proposes a transparent robust alignment solution based on contributors’ reported judgments of what ought to be more recommended on YouTube.[9]
So, are we dropping the ball?
There are few systems today as pervasive as recommendation AI on collective psyche.
Here are a few bottlenecks that might make improvements more challenging than they initially appear:
Overall, I'm tempted to say that yes, we are probably dropping the ball.
Additional statistics on YouTube can be found here or in this book.
Bad recommendation AIs → Bad epistemic → Misinformed and misaligned politician → catastrophic decisions with respect to transformative AI.
This would be one possible causal chain towards more X-Risks.
"it is not isolation from opposing views that drives polarization but precisely the fact that digital media bring us to interact outside our local bubble. When individuals interact locally, the outcome is a stable plural patchwork of cross-cutting conflicts. By encouraging nonlocal interaction, digital media drive an alignment of conflicts along partisan lines, thus effacing the counterbalancing effects of local heterogeneity. The result is polarization, even if individual interaction leads to convergence." From a paper that tries to modelize the polarization dynamics.
"The result of that, it turns out that what gets the most comments is really divisive, outrageous stuff, especially stuff that provokes political anger." - source
Another summary of the Facebook leak is available on wikipedia.
"Meta uses engagement-based algorithmic systems to power Facebook's news feed, ranking, recommendation and groups features, shaping what is seen on the platform. Meta profits when Facebook users stay on the platform as long as possible, by selling more targeted advertising. The display of inflammatory content – including that which advocates hatred, constituting incitement to violence, hostility and discrimination – is an effective way of keeping people on the platform longer. As such, the promotion and amplification of this type of content is key to the surveillance-based business model of Facebook." (source)
This raises the question of what is important. Maybe you are a libertarian and would say, "What is important is what people choose to watch." But I think that even with this definition, there is a difference between preferences and volition, the latter being chosen in a much more mindful way. Better recommendation AI could enable users to watch content they truly want to watch upon reflection, which is very different from merely optimizing for immediate preferences. And if, upon reflection, people genuinely want to eat fast food, then so be it.
For example, Pol.is, an opinion mapping tool that uses machine learning to identify areas of consensus and disagreement among participants. Unlike traditional recommendation AIs that can amplify polarization, Pol.is is designed to highlight points of agreement, thus fostering more constructive debate, and discussions are organized in several phases (proposal, discussion, reflection, decision), allowing for an orderly and transparent progression of debate.
Taiwan's democratic renaissance over the past decade stands out as a rare success story in a world where many democracies have faltered. The catalyst for this transformation was the 2014 Sunflower Student Movement, which sparked a shift from an imperfect democracy vulnerable to Chinese influence and corruption to a model of democratic governance.
Central to this evolution was Taiwan's commitment to democratizing the digital sphere. Under the leadership of figures like Audrey Tang, who became Digital Minister, Taiwan invested heavily in democratic digital technologies. These initiatives included innovative reforms to enhance government transparency and citizen participation through digital tools.
Sunflower in English
Numerous mathematical, sociological and philosophical problems have been identified by the project, some of which are well defined, and have been argued to be central to any collaborative AI alignment problem.
Maybe, another organisation that could qualify could be The Mozilla Foundation: Mozilla has been advocating for transparency and ethical approaches in technology, including research into how recommendation systems work. They have conducted studies on YouTube's recommendation algorithm and its role in promoting harmful content. And they have also launched initiatives like the YouTube Regrets project, which collects stories from users who were led down undesirable recommendation "rabbit holes."
Hint: This does not only apply to ML models