Inspired by a recent comment, a potential AI movie or TV show that might introduce good ideas to society, is one where there are already uploads, LLM-agents and biohumans who are beginning to get intelligence-enhanced, but there is a global moratorium on making any individual much smarter.
There's an explicit plan for gradually ramping up intelligence, running on tech that doesn't require ASI (i.e. datacenters are centralized, monitored and controlled via international agreement, studying bioenhancement or AI development requires approval from your country's FDA equivalent). There is some illegal research but it's much less common. i.e the Controlled Takeoff is working a'ight.
If it were a TV show, the first season would mostly be exploring how uploads, non-sentient-LLMs, enhanced humans and regular humans coexist.
Main character is an enhanced human, worried about uploads gaining more political power because there are starting to be more of them, and research to speed them up or improve them is easier.
Main character has parents and a sibling or friend who are choosing to remain unenhanced, and there is some conflict about it.
By the end of season 1, there's a subplot about illegal research into rapid superintelligence.
I think this sort of world could actually just support a pretty reasonable set of stories that mainstream people would be interested in, and I think would be great to get the meme of "rapidly increasing intelligence is dangerous (but, increasing intelligence can be good)" into the water.
I think I'm imagining "Game of Thrones" vibes but it could support other vibes.
I have not looked into these details enough to have an opinion, but, I think a lot of US institutions work via a mix of legal rules and implicit norms, and my sense is Trump was doing a lot of violating the norms that made legal rules workable
The worlds I was referring to here were worlds that are a lot more multipolar for longer (i.e. tons of AIs interacting in a mostly-controlled-fashion, with good defensive tech to prevent rogue FOOMs). I'd describe that world as "it was very briefly multipolar and then it wasn't" (which is the sort of solution that'd solve the issues in Nice-ish, smooth takeoff (with imperfect safeguards) probably kills most "classic humans" in a few decades.
1I doubt that it's correct. Suppose that Agent-4 solves alignment to itself. If Agent-4-aligned AIs gain enough power to destroy the world, then any successor would also be aligned to Agent-4 or to a compromise including Agent-4's interests (which could actually be likely to include the humans' interests).
Sounds like this scenario is not multipolar? (Also, I think the crux is solveable, see the linked post, but solving it requires hitting particular milestones quickly in particular ways)
I am not sure whether AI rots the agency of the people whose decisions are actually important.
Why not?
(my generators for this belief: my own experience using LLMs, the METR report on downlift suggesting people are bad at noticing when they're being downlift, and general human history of people gravitating towards things that feel easy and rewarding in the moment)
Both? My impression was they (Redwood in particular but presumably also OpenAI and Anthropic) expected to be using a lot of AI assistance along the way.
But, when I said "constraints" I meant "solving the problem requires some set of criteria", not "applying constraints to the AI" (although I'd also want that).
Where, constraints would be like "alignment is hard in a way that specifically resists full-handoff and it requires a philosophically-competent human in the loop till pretty close to the end." (and, then specifically operational-detail-constraints like "therefore, you need to have a pretty good map of which tasks can be delegated")
1Nod.
My main project thread for the past 2 years has been mostly aiming at Get a Lot of Alignment Research Done Real Fast (in line with my beliefs/taste about what that requires). This is the motivator for the Feedbackloop-first Rationality project, and is also a driver for my explorations into using LLMs for research (where I'm worried specifically about phrases like "full handoff" because of the way it seems like LLM-use subtly saps/erodes agency and direct you towards dumber thoughts that more naturally 'fit' into the LLM paradigm. But I'm also excited about approaches for solving that).
But I'm focused for this year on "wake everyone up."
Corrigibility and CEV are trying to solve separate problems? Not sure what your point is here; agreed on that being one of the major points of CEV.
If every country/person was building CEV, it wouldn't be particularly scary (from a misuse standpoint). Whereas if every country is focused on corrigibility, there will be a phase where unilateral actors can do bad stuff you need to worry about.
1I'm not sure how you're contrasting this with the point I was making.
No, it's me expressing disagreement with your reasoning for "A few of these are, if somewhat unprecedented, not really institutional erosion, because they have a legitimate constitutional basis."
because, a constitutional basis is necessary but not sufficient (because soft cultural norms are also important)
(But, this is an area I have not looked into enough to have a strong belief about the object level claims, just objecting to your reasoning as sufficient to prove the point you wanted to make)