Posts
Wikitag Contributions
Implicit in my views is that the problem would be mostly resolved if people had aligned AI representatives which helped them wield their (current) power effectively.
Can you make the case for this a bit more? How are AI representatives going to help people prevent themselves becoming disempowered / economically redundant? (Especially given that you explicitly state you are skeptical of "generally make humans+AI (rather than just AI) more competitive").
Mandatory interoperability for alignment and fine-tuning
Furthermore, I don't really see how fine-tuning access helps create AI representatives. Models are already trained to be helpful and most people don't have very useful personal data that would make their AI work much better for them (that can't be put in context of any model).
The hope here would be to get the reductions in concentration of power that come from open source
The concentration of power from closed source AI comes from (1) the AI companies' profits and (2) the AI companies having access to more advanced AI than the public. Open source solves (1), but fine-tuning access solves neither. (Obviously your "Deploying models more frequently" proposal does help with (2)).
Yeah that would be great!
I appreciate the effort to avoid the usual downsides of feeds. But your arguments that this feed will be good actually don't persuade me. I already spend more time than I endorse on LessWrong, especially since the introduction of quick takes.
Does anyone understand the real motivation here? Who at Anthropic makes the call to appoint a random CEO who (presumably) doesn't care about x-risk, and what do they get out of it?
This is the best proposal I've read for making progress on the question of "What should we point the ASI at?"
Are humans such that they can be brought through a series of arguments and endorse all of the premises in the arguments and end up with radically different moral views
I don't feel quite satisfied with this operationalization. It could Goodharted (especially by an ASI) by finding adversarial inputs to humans.
I also think someone could decide that their meta-preference is to endorse their unconsidered preferences and that the act of thinking about moral arguments is actually against their morality. This may sound absurd but probably accurately describes many people's responses to drowning child thought experiments. Most people don't act as if they have utility functions and I don't see why people should be required to adopt one. To the extent that it matters a lot what people think about their preferences upon reflection, they will probably notice the inconsistencies in their preference once they start Dutch-booking themselves by manifesting inconsistent preferences with the help of ASI.
kinds of questions we know how to solve, and could solve with the help of AI
I predict that once people can talk to ASIs, many will quickly see the bizareness of the realist view and a lot more people will start saying "I don't know what you're talking about, can you please make this clearer?".
Uhh yeah sorry that there hasn't been a consistent approach. In our defence I believe yours in the only complex moderation case that PauseAI Global has ever had to deal with so far and we've kinda dropped the ball on figuring out how to handle it.
For context my take is that you've raised some valid points. And also you've acted poorly in some parts of this long running drama. And most importantly you've often acted in a way that seems almost optimised to turn people off. Especially for people not familiar with LessWrong culture, the inferential distance between you and many people is so vast that they really cannot understand you at all. Your behavior pattern matches to trolling / nuisance attention seeking in many ways and I often struggle to communicate to more normie types why I don't think you're insane or malicious.
I do sincerely hope to iron this out some time and put in place actual systems for dealing with similar disputes in the future. And I did read over the original post + Google doc a few months ago to try to form my own views more robustly. But this probably won't be a priority for PauseAI Global in the immediate future. Sorry.
Meta-level comment now that this has been retracted.
Anthropic's safety testing for Claude 4 is vastly better than DeepMind's testing of Gemini. When Gemini 2.5 Pro was released there was no safety testing info and even the model card that was eventually released is extremely barebones to compared to what Anthropic put out.
DeepMind should be embarrassed by this. The upcoming PauseCon protest outside DeepMind's headquarters in London will focus on this failure.
1I think the reason why dreams of ideas are generally not that useful is because they are usually "empty". The specific details of ideas are so important that if you haven't actually figured out what those details are, then your prior should be that, when you try to flesh it out, the idea is probably not going to be much better than what you would come up with 5 seconds of wishful thinking.
Anthropic is reportedly lobbying against the federal bill that would ban states from regulating AI. Nice!