The section on Chevron Overturned surprised me. Maybe I'm in an echo chamber, but my impression was that most legal scholars (not including the Federalist Society and The Heritage Foundation) consider the decision to be the SCOTUS arrogating yet more power to the judicial branch, overturning 40 years of precedent (which was based on a unanimous decision) without sufficient justification.
I consider the idea that "legislators should never have indulged in writing ambiguous law" rather sophomoric. I don't think it's always possible to write law that is complete, unambiguous, and also good policy. Nor do I think Congress is always the best equipped to do so. I don't fully trust government agencies delegated with rulemaking authority, but I have much less trust that a forum-shopped judge in the Northern District of Texas is likely to make better-informed decisions about drug safety than the FDA.
FWIW, I haven't really thought much about Loper as it relates to AI, tech, and crypto specifically. The consequences of activist judges versus the likes of the DOJ, CDC, FDA, and EPA are mostly what come to mind. Maybe it's attention bias given recent SCOTUS decisions versus more limited memory of out-of-control agencies but I feel uneasy tilting the balance of power toward judicial dominance.
[Carl Shulman] is assuming normality where he shouldn’t, and this is one of the key places for that. It is a vision of AGI without ASI
This is valid exploratory engineering, which assumes some capabilities and considers what can be done with at least those capabilities. There is no implication that this is what will be done, or that capabilities won't be much greater. We can still conclude that what can be done with merely these capabilities will remain an option given greater capabilities. Forecasting of optionality, not of actuality.
On the one hand, I also wish Shulman would go into more detail on the "Supposing we've solved alignment and interpretability" part. (I still balk a bit at "in democracies" talk, but less so than I did a couple years ago.) On the other hand, I also wish you would go into more detail on the "Humans don't benefit even if you 'solve alignment'" part. Maybe there's a way to meet in the middle??
I also wish you would go into more detail on the "Humans don't benefit even if you 'solve alignment'" part.
My own answer to this is that humans aren't secure, and AI will exacerbate the problem greatly by helping the offense (i.e. exploiting human vulnerabilities) a lot more than the defense. I've focused on philosophy (thinking that offense merely requires training AI to be persuasive, while defense seems to require solving metaphilosophy, i.e., understanding what correct philosophical reasoning consists of), but more recently realized that the attack surface is so much bigger than that. For example humans can fall in love (resulting in massive changes to one's utility function, if you were to model a human as a utility maximizer). It will be straightforward to use RL to train AI to cause humans to fall in love with them (or their characters), but how do you train AI to help defend against that? Would most humans even want to defend against that or care enough about it?
So even with alignment, the default outcome seems to be a civilization with massively warped or corrupted human values. My inner Carl wants to reply that aligned and honest AI advisors will warn us of this and help us fix it before it's too late, maybe by convincing policymakers to pass regulations to prevent such "misuse" of AI? And then my reply to that would be that I don't see how such regulations can work, policymakers won't care enough, it seems easier to train AI to attack humans than to make such predictions in a way that is both honest/unbiased and persuasive to policymaker, AI might not have the necessary long-horizon causal understanding to craft good regulations before it's too late.
Another possibility is that alignment tax is just too high, so competitive pressures erode alignment even if it's "solved" in some sense.
only to find out it is censored enough I could have used DALL-E and MidJourney.
Last "censoring" of Stable Diffusion was done via the code and could've been turned off via 2 lines of code change. Was it done other way this time?
Apparently. Stability claimed "numerous safeguards" starting from the data onwards, which cannot simply be disabled by 1 line of Python if nsfw(generated image) then error, as I recall, and while I can't find any specifics easily about what all of the anti-NSFW countermeasures were, I don't see anyone claiming to have trivially beaten them. If they did something like filter the training data even more heavily to eliminate all kinds of nudity & suggestive imagery & porn, then ignorance may be baked into the model's final concepts so heavily you might as well not even bother with SD3 and work on something else.
1Arena is less kind to DeepSeek, giving it an 1179, good for 21st and behind open model Gemma-2-9B.
DeepSeek Coder V2 is a coding-oriented model. To evaluate how it is doing at its targeted specialization, go to https://chat.lmsys.org/?leaderboard, but then make sure to switch to coding.
Then you'll see that it is at position 10 (and with the Arena ways to group ranks, it is sharing rank 5 with models like Claude 3 Opus and GPT-4 January preview while being somewhat behind them; it is slightly above a number of formidable models, such as Gemini 1.5 Flash, an older version of Gemini 1.5 Pro, Claude 3 Sonnet, and Gemma-2-27B).
It does look pretty formidable, and a clear leader among open weight models in coding tasks.
The ‘answer engine’ idea is great, and occasionally I find it the right tool for the right job although mostly I end up at either the Google Search or Claude Sonnet ends of the spectrum.
It's surprisingly rare that Perplexity can answer a question that google can't for me.
What I really need from AI search is usually far better serviced by the little known and unspectacular exa.ai.
Your writing about how/when/why having all these AIs around goes wrong is exceptionally coherent and sensical and buyable IMO. Do you have much opportunity to preach outside the choir? Do you have late-night TV skills? I think you could have a much much larger platform. And actually get a substantial and correct message across. (Or were you doing lobbying.)
Chevron deference is no more. How will this impact AI regulation?
The obvious answer is it is now much harder for us to ‘muddle through via existing laws and regulations until we learn more,’ because the court narrowed our affordances to do that. And similarly, if and when Congress does pass bills regulating AI, they are going to need to ‘lock in’ more decisions and grant more explicit authority, to avoid court challenges. The argument against state regulations is similarly weaker now.
Similar logic also applies outside of AI. I am overall happy about overturning Chevron and I believe it was the right decision, but ‘Congress decides to step up and do its job now’ is not in the cards. We should be very careful what we have wished for, and perhaps a bit burdened by what has been.
The AI world continues to otherwise be quiet. I am sure you will find other news.
Table of Contents
Language Models Offer Mundane Utility
Guys. Guys.
I get it, because I already get it. But who is the customer here? I would have spent a few extra words to ensure people knew this was an AI and LLM thing?
Anthropic’s marketing problem is that no one knows about Claude or Anthropic. They do not even know Claude is a large language model. Many do not even appreciate what a large language model is in general.
I realize this is SFO. Claude anticipates only 5%-10% of people will understand what it means, and while some will be intrigued and look it up, most won’t. So you are getting very vague brand awareness and targeting the congnesenti who run the tech companies, I suppose? Claude calls it a ‘bold move that reflects confidence.’
Language Models Don’t Offer Mundane Utility
David Althus reports that Claude does not work for him because of its refusals around discussions of violence.
Once again, where are all our cool AI games?
Summarize everything your users did yesterday?
If you want it to be accurate in the ‘assume this is correct and complete’ sense then no, that’s not going to happen soon. The bar for useful seems far lower, and far more within reach. Right now, what percentage of important user stories are you catching? Almost none? Now suppose the AI can give you 50% of the important user stories, and its items are 80% to be accurate. You can check accuracy. This seems highly useful.
In general, if you ask what the AI cannot do, you will find it. If you ask what the AI can do that is useful, you will instead find that.
Similarly, here (from a few weeks ago) is Google’s reaction on the question of various questionable AI Overviews responses. They say user satisfaction and usage was high, and users responded by making more complex queries. They don’t quite put it this way, but if a few nonsense questions like ‘how many rocks should I eat’ generate nonsense answers, who cares? And I agree, who cares indeed. The practical errors are bigger concerns, and they are definitely a thing. But I am often happy to ask people for information even when they are not that unlikely to get it wrong.
Thread asks: What job should AI never be allowed to do? The correct answer is there. Which is, of course, ‘Mine.’
Opinion piece suggests AI could help Biden present himself better. Um… no.
Man in the Arena
Arena results are in. The top is not where I expected.
Claude Sonnet is also slightly ahead of GPT-4o on Coding, with a big gap from GPT-4o to Gemini, and they are tied on the new ‘multi-turn.’ However GPT-4o remains on top overall and in Hard Prompts, in Longer Query and in English.
Claude Opus also underperforms on Arena relative to my assessment of it and eagerness to use it. I think of Sonnet as the clear number one model right now. Why doesn’t Arena reflect that? How much should we update on this, and how?
My guess is that Arena represents a mix of different things people evaluate, and that there are things others care about a lot more than I do. The reports about instruction handling and math matter somewhat on the margin, presumably. A bigger likely impact are refusals. I have yet to run into a refusal, because I have little reason to go to places that generate refusals, but GPT-4o is disinclined to refuse requests and Claude is a little tight, so the swing could be substantial.
We are talking about tiny edges among all the major offerings in terms of win percentage. Style plausibly also favors GPT-4o among the voters, and it is likely GPT-4o optimized on something much closer to Arena than Claude did. I still think Arena is the best single metric we have. We will have to adjust for various forms of noise.
Another ran,ing system here is called Abacus, Teortaxes notes the strong performance of deepseek-coder-v2, and also implores us to work on making it available to use it as competition to drive down prices.
I too find these ratings suspect. In particular the big drop to Gemini 1.5 Pro does not pass my smell test. It is the weakest of the big three but this gap is huge.
Arena is less kind to DeepSeek, giving it an 1179, good for 21st and behind open model Gemma-2-9B.
And as another alternative, here is livebench.ai.
These other two systems give Claude Sonnet 3.5 a substantial lead over the field.
That continues to match my experience.
Fun with Image Generation
Claude provides map of different types of shots and things I can enter for my prompt.
Andrej Karpathy uses five AI services to generate thirty seconds of mildly animated AI pictures covering the first 28 seconds of Pride and Prejudice. I continue to not see the appeal of brief panning shots.
Also given the slow news week I had Claude set up Stable Diffusion 3 for me locally, which was a hilarious odyssey of various technical failures and fixes, only to find out it is censored enough I could have used DALL-E and MidJourney. I hadn’t thought to check. Still, educational. What is the best uncensored image model at this point?
Deepfaketown and Botpocalypse Soon
AI submissions on university examinations go undetected 94% of the time, outperform a random student 83.4% of the time. The study took place in Summer 2023 and minimal prompt engineering was used. If you are a university and you give students take home exams, you deserve exactly what you get.
This is not obviously that good a rate of going undetected? If you take one midterm and one final per class, three classes per term for eight terms, that’s 48 exams. That would give you a 95% chance of getting caught at least once. So if the punishment is severe enough, the 6% detection rate works. Alas, that is not what detected means here. It simply means any violation of standard academic policy. If the way you catch AI is the AI violates policy, then that number will rapidly fall over time. You could try one of the automated ‘AI detectors’ except that they do not work.
Nonsense chart found in another scientific journal article. As in complete gibberish. Whatever our ‘peer review’ process does not reliably detect such things.
I’ve speculated about this and John Arnold has now tweeted it out:
There are several things that would have to happen. First, there would need to be good enough AI image generation that people could not tell the difference even under detailed analysis. This is a very high bar, much harder than passing an initial eye test. Also, how do you fake information that is not available to the model, such as intimate details? Second, people would have to reason through this and adjust enough to not react. I do expect some reduction in impact as cultural norms shift.
They Took Our Jobs
Hard work in Minecraft, as hundreds of AI agents do their tasks, file their results in a Google sheet, a journalist AI agent reviews and writes a report and then the agents update their plans.
We are sufficiently early that the ways we get agents to work together are ‘create facsimiles of things humans do.’ Last week we had virtual water coolers. There are presumably much better ways to do this, but it is like the bitter lesson, in the sense that doing anything at all is going to get you interesting results and so what if your method is obviously horribly inefficient.
The Art of the Jailbreak
Here we have ‘unleashing Claude 3.5 Sonnet as a hacker.’
Or as they puts it here:
Pliny seems to be getting better at jailbreaking a lot faster than the AI companies are getting better at preventing jailbreaks.
He does however seem a little confused about the implications? Or perhaps not, it is confusing. The important thing is that every model so far can and will be jailbroken, fully, by anyone who cares enough to do so.
Pliny could not, under SB 1047, cause ‘a shutdown of the entire AI industry’ partly because no or almost no existing models are even covered under SB 1047 (oh, sure, that). But also because the whole point is that you have to act as if such jailbreaks exist until such time as one can make them not exist.
Thus, Pliny’s skillset is highly useful for safety, exactly because it lets you test the fully jailbroken model.
If you give people access to an open weights model, you give them access to anything you can create from there via a reasonable amount of fine tuning, which includes things like ‘nullify all safety fine-tuning’ and ‘fill in any knowledge gaps.’
Similarly, For closed models, for all practical purposes, what you are releasing when you give people access to a model is the jailbroken version of that model. You have to test the capabilities after the safety restrictions get bypassed, or you have to actually create safety restrictions that are a lot harder to bypass.
Until then, yes, when METR or the UK tests an AI model, they should test it via (1) jailbreaking it then (2) testing its capabilities. And if that turns out to make it too dangerous, then you do not blame that on Pliny. You thank them.
Get Involved
Free NYC space for tech events and related happenings.
Anthropic is accepting proposals for third party model evaluations.
Yoshua Bengio looking for people to work with him on Bayesian approaches to AI safety.
Anthropic recruiting happy hour on July 23… in Vienna?
80,000 Hours is running a census of everyone interested in working on reducing risks from AI, and asked me to pass it along. This census will be used to help connect organisations working to advance AI safety with candidates when they’re hiring so that more talent can be directed to this problem. They say they are keen to hear from people with a wide range of skill sets — including those already working in the field.
Introducing
OpenAI gets Time magazine to sign up their content.
Etched introduces Sohu, a chip that is locked into only using the transformer architecture and discards everything devoted to other functionalities. They claim this makes it vastly cheaper and faster than Nvidia chips. I don’t know enough about hardware to know how seriously to take the claims. The first obvious question, as is often the case: If true, why aren’t more people talking about it?
Open weights model Gemma 2 released by DeepMind, sizes 9B and 27B. Gemma 27B is now the highest rated open model on Arena, beating Llama-70b outright.
They also are releasing the full 2 million token context window for Gemini 1.5 Pro and enabling code execution for 1.5 Pro and 1.5 Flash.
From the men who host the Arena, introducing RouteLLM. Mix and match various LLMs via data augmentation techniques.
ElevenLabs offers Iconic Voices feature, setting up Hollywood star voices for you.
In Other AI News
Pixel 9 to include a feature called ‘Pixel Screenshots.’ Unlike Microsoft’s ‘always on and saving everything in plaintext,’ here you choose to take the screenshots. This seems like The Way.
Quiet Speculations
Amanda Askell points out that if you can have one AI employee you can have thousands. That doesn’t mean you know what to do with thousands. There are a lot of tasks and situations that have good use for exactly one. Also Howard notes that costs scale with the virtual head count.
AI Snake Oil’s Narayanan and Kapoor proclaim scaling will run out and the question is when. They argue roughly:
This seems like a conflation of ‘will run out before AGI’ with ‘might run out before AGI.’ These are great arguments for why scaling might run out soon. And of course scaling will eventually run out in the sense that the universe is headed for heat death. They do not seem like good arguments for why scaling definitely will run out soon. Thus, when they say (as Robin Hanson quotes):
This is a confusion between reasonable doubt and actual innocence. One frequently should ‘lack confidence’ in something without having confidence in its negation.
Also I strongly disagree with their model of point three. It is true that the models are already capable enough for many highly valuable use cases, where becoming faster and cheaper will be more useful on the margin than making the model smarter. However there are also super valuable other things where being smarter is going to be crucial.
Justis Mills finds MatMul potentially promising as a transformer alternative, but notes it is untested on larger models and the tests it did run were not against state of the art, and that even if it is superior switching architectures is at best slow.
Robin Hanson’s latest cold water throwing on AI progress:
Something is impressive largely if it is some combination of:
A lot of advances in AI indicate that AI in general and this actor in particular have higher capability and skill, and thus indicate some combination of current and future usefulness. AI is on various exponentials, so most things that impress in this way are impressive because of future use, not present use. And the future is unevenly distributed, so even the things that are useful now are only useful among a select few until the rest learn to use them.
Is there a conflict between impressive and useful? Yes, sometimes it is large and sometimes it is small.
New Paper: AI Agents That Matter.
As is often the case with papers, true statements, I suppose someone had to say it:
Noah Smith endorses Maxwell Tabarrok’s critique of Acemoglu’s recent paper. Noah does an excellent job crystalizing how Acemoglu went off the rails on Acemoglu’s own terms. How do you get AI to both vastly increase inequality and also not create economic growth? It helps to, for example, assume no new tasks will be created.
Here is a new version of the not-feeling-the-AGI copium, claiming that LLMs that are not ‘embodied’ cannot therefore have tacit knowledge, I believe through a circular definition and ‘this is different from how humans work’ but in any case the core claim seems obviously false. LLMs are excellent at tacit knowledge, at picking up the latent characteristics in a space. Why would you think Humean knowledge is harder for an LLM rather than easier? Why would you similarly think Hayekian detail would be available to humans but not to LLMs? All the good objections to an LLM having either of them applies even more so to humans.
Andrej Karpathy continues to pitch the Large Language Model OS (LMOS) model.
I do not think this is going to happen. I do not think this would provide what people want. I want my operating system to be reliable and predictable and fast and cheap. Might I use an LLM to interface with that operating system? Might many people use that as their primary interaction form? I can see that. I cannot see ‘context window instead of RAM’ are you insane? Or are you looking to be driven that way rapidly?
The Quest for Sane Regulations
The bad faith attacks and disconnections from reality on SB 1047 continue, including an attempt from Yann LeCun to hit bill consultant Dan Hendrycks for ‘disguising himself as an academic’ when he is a heavily cited academic in AI.
Scott Weiner has responded to some such attacks by YC and a16z in a letter, in which he bends over backwards to be polite and precise, the exact opposite of a16z’s strategy.
I am no longer even disappointed, let alone saddened or infuriated, by those who repeatedly double down on the same false claims and hysteria. It is what it is. Their claims remain false, and SB 1047 keeps passing votes by overwhelming margins.
In other Scott Weiner news, the same person was also behind SB 423, which will now hopefully greatly accelerate housing construction in San Francisco. I have seen zero people who think Weiner is out to get them notice their confusion about this.
Chevron Overturned
I’m going to cover Loper and Chevron generally here, not only the AI angle.
Is Loper the right decision as a matter of law and principle? I am pretty sure that it is.
Am I overall happy to see it? Yes I am.
One must always beware mood affiliation.
If you are rooting against bureaucracy being functional, and for breakdowns in the government, that seems like the wrong thing to root for. You do not want to be ‘against bureaucracy.’ You want to be against abuse of power, against capricious rules, against overreach. You want to be for state capacity and good government. It is reasonable to worry that this could cause a lot of chaos across many fronts.
William Eden points out that judges are indeed experts at figuring out who has jurisdiction over things and settling disputes. I’d also add that this was already necessary since overreach was common either way. The difference at equilibrium is the barriers should be clearer.
Certainly many hysterical people did poorly here, but also reminder that people crying wolf in the past does not provide that much evidence regarding future wolves beyond ignoring their warnings:
Brenan Carr has several good points. Major questions are the purview of the major questions doctrine, which has not changed. He says (credibly, to me) that the lion’s share of Chevron cases are challenges to new regulatory requirements imposed on private citizens or business. And he points out that Chevron was never how law otherwise works, whereas Loper very much is.
However, be careful what you wish for, for AI, for startups and in general.
As Leah Libresco Sargeant replies, Congress is now rather slow on the uptake, and highly dysfunctional. Even if ‘everyone agrees’ what the obvious fix is (see for example the IRS and software engineers being amortized over years) that does not mean Congress will fix it. Indeed, often ‘you want this fixed more than I do’ means they hold out for ‘a deal.’
Adam Thierer (RSI) discusses what to expect after Loper overturned Chevron.
If the courts challenge making rule of law impractical, but allow you to instead do rule of man and via insinuation and threats, that’s what you will get.
Is the response going to be Congress stepping up and making good laws again?
This is why Ally McBeal’s therapist has her laugh track button.
This seems very right, and one must be realistic about what happens next:
As she then points out, Congress lacks sufficient resources to actually do its job. That is one reason it hasn’t been doing it. There are also others. So this is great if it got Congress to do its job and give itself the resources to do so, but even if that eventually happens, the transition period quite plausibly is going to suck.
Those ‘good laws’ plausibly only get harder if you force everything to be that much more concrete, and you strip away the middle ground via Chevron. And Congress was struggling a lot even on the easiest mode.
Charlie Bullock discusses Chevron and AI at Institute for Law & AI. His assessment is this makes it harder to regulate AI using existing authority, same as everything else. A common refrain is that ‘existing law’ is sufficient to regulate AI. A lot of that ‘existing law’ now is in question and might no longer exist with respect to this kind of extension of authority that was not anticipated originally (since Congress did not forsee generative AI), so such arguments are weakened. In which particular ways? That is less clear.
One thing I have not heard discussed is whether this will encourage much broader grants of rulemaking authority. If every ambiguous authority resolves against the agency, will Congress feel the need to give ‘too much’ authority? Once given, we all know that the regulators would then use it. Perhaps the ambiguity was doing work.
You can think of hard vs. soft law, or careful vs. blunt law, or good vs. bad law, or explicit vs. implicit law, or rule of law vs. rule of man (vs. rule by machine).
The option you will not have, not for very long, is no law. If you ban hard you get soft, if you punish explicit you get implicit, if you defeat careful you get blunt, if you fight good you end up with bad. If rule of law is unworkable, you have two options left, which one is it going to be?
Without Chevron, and with certain people fighting tooth and nail against any attempt to do precise well-considered interventions and also the general failures of Congress, there is less room (as I understand it) for improvised ‘medium’ solutions, and the solution types we would all prefer seem more likely to be blocked.
Thus I fear by default Adam is right on this on the margin. That also means that those most vulnerable to government soft power have to tiptoe around such threats, and those less vulnerable have no idea how to comply and instead hope they don’t trigger the hammer, which is not the way to do things safely.
My default guess is that things do not change so much. Yes, it will be a mess in many ways, but all the talk of big disasters and opportunities will prove overblown. That is usually the safe default. As I understand the ruling, you can still delegate authority, the only difference is that Congress has to explicitly do that. Mostly I’d presume various workarounds mostly suffice.
Deb Raji disagrees and sees this as gutting our ability to respond because we were entirely dependent on rulemaking authority, and the flexibility to respond as circumstances change.
Balaji of course calls this ‘Chevron Dominance’ and says ‘technology is about to accelerate.’ It’s funny. He thinks ‘Congress did not give the SEC the authority to relegate crypto’ as if being on a blockchain should make you immune to existing laws. The SEC has authority over securities. You made new securities. That’s on you. But more generally, he is saying ‘regulators just got disarmed’ and that everyone’s now free to do what they want. ‘I can already feel the T-levels across tech increasing,’ he says.
As another example, Austen Allred has a thread saying this ‘may be the most impactful thing to happen to startups in a long time,’ full of some very choice words for Chevron and the SEC. At some point that counts as supreme restraint. And certainly not being told how to comply with the law is infuriating.
I notice a clear pattern. For some people, no matter what It might be, It is always A Big Deal. Any little movement changes everything. Miami bans lab-grown meat? RIP Miami. California says giant frontier models have to do paperwork? RIP startup ecosystem. And it works in the other direction, too, Chevron is gone so LFG. They talk about lots of other aspects of a business the same way.
Scott Adams explained back in 2016 why Trump talks this way, it exerts maximum leverage until and unless people properly adjust for it. Similarly, everyone in crypto is always super hyped about whatever it is, and how it is changing everything. Which it isn’t.
Justin Slaughter thinks this is a sea change. You won’t be able to extend your authority to new areas as they arise without Congress approving, an increasingly tough ask. And he also warns of the shift to enforcement actions.
When everyone says ‘oh great, now they will have to tell us the rules or else let us build, we can do all sorts of cool startups now!’ I sincerely hope that it works that way. I fear that in practice it is the other way. For crypto in particular I think the SEC is on solid ground from a technical legal perspective, and people should not get overexcited.
Here is another illustration of the problem, from Matt Bruenig and Matthew Zeitlin:
I agree we should aspire to what the tech people want here. We should demand it, to the extent possible, that we be told what is legal and what is illegal.
That is not, alas, how our system works, or how it fully can work. The regulators are not there to decide in advance exactly what the rule is for you.
In particular, they are not there to help you tippy-toe up to the edge, figure out exactly how to pull off your regulatory arbitrage, and then stand there powerless to do anything because technically they said what you are doing was acceptable and you don’t have to play by the same rules as Megacorp. Or, alternatively, to give you an opinion, then you use that to sue them. Also no fun from their side.
The good news from that perspective is this sets off a bunch of lawsuits. Those lawsuits provide clarity. The bad news is that this discourages rule making in favor of vague indications and case by case policy. That is not what startups want.
The Week in Audio
Carl Shulman spends over four hours on 80,000 hours talking about the economy and national security after AGI, and it is only part 1. A lot of the content is similar to Carl’s talk with Dwarkesh Patel last year.
I continue to feel like Carl is spending a lot of time on, maybe not the wrong questions, but not the questions where I have uncertainty.
Yes, there is a ton of energy available and in some theoretical sense we could do all the things. Yes, replication can if done efficiently happen fast. Yes, AGI could solve robots and do all the things. We know all that. The vision is ‘if we have lots of super capable AIs that do things humans want and coordinate to do that in ways that are good for humans, we would have all the things and solve so many problems,’ and yeah, fine, we agree.
Indeed, the central theme of this podcast is ‘people have this objection, but actually if you look at the physical situation and logic behind it, that objection matters little or is rather dumb’ and indeed, Carl is basically always right about that, most of the objections people make are dumb. They are various forms of denying the premise in ways more basic than where Carl ignores the implications of the premise.
They first goes through six core objections to Carl’s vision.
So I indeed find those objections unconvincing. But the obvious seventh objection is missing: Won’t these AGIs very quickly have control over the future? Why would all this energy get spent in ways that benefit humans, even if you do ‘solve alignment’? And what makes you think you can solve that while charging forward?
I can’t get past this implicit (and often explicit) idea that something has to go actively wrong for things to end badly. The ‘risk of accidental trouble, things like a rogue AI takeover,’ instead of thinking that in a world transformed every few months where AIs do all the work and are more capable and efficient than us in every way us staying in charge seems pretty unlikely and weird and hard to pull off.
In the discussion of inequality and income, Carl says there will be tons of pressure from people to redistribute some of this vastly greater wealth, and plenty to go around, so there is no need to worry. Why would we assume this pressure impacts what happens? What is this ‘in democracies’? Why should we expect such things to long endure in these scenarios? Again, aren’t we assuming some very weirdly narrow range of AGI capabilities but not further capabilities for any of this to make sense?
The discussion of economists starts with Carl agreeing that ‘they say no way’ and yeah, they say that.
Then he goes over Baumol effect arguments, which are dumb because these AGIs can do all the things, and even if they can’t you can change the basket to work around the missing elements.
Or they deny robots can exist because robotics is unsolvable, which means they should not interrupt the people solving it, and also Carl points out so what, it would ultimately change little and not slow things down that much even if robots was indeed unsolvable because literal physical humans could be the robots with AIs directing them. And that’s largely good enough, because this whole scenario is actually being highly unimaginative.
What about input shortages especially for semiconductors? Carl answers historically rapid growth is common. I would add that with AGI help on this front too it would get a lot easier to go faster.
Carl points out that standard economic models actually very much do imply super rapid economic growth in these spots. Economists mostly refuse to admit this and instead construct these models where AI is only this narrow thing that does particular narrow tasks and make the assumptions that drive their absurd conclusions.
Won’t we be slow to hand over decision making to AIs? Carl points out that if the incentives are strong enough, we will not be that slow.
Why are economists dropping this ball so badly? They speculate about that, Carl points out some Econ 101 standard intuitions that stand in the way, and they are used to bold claims like this being wrong. And the economists expect everything to be gradual and ‘economic normal,’ and don’t get that this won’t hold.
They then spend an hour on the moral status of AIs. It is so weird to build up this whole model assuming the humans stay in charge, only then to notice that 99.999% of the intelligences in this world, that are more capable than humans, are not humans and may have moral standing, and then offhand say ‘well in these scenarios we have solved alignment and interpretability, so…’. And then they talk about these minds having open ended goals and wanting to survive and taking on risk and so on, and yes during this hour they notice the possibility of AI ‘domination.’
There is a part 2 coming, and it looks like it will address these issues a nonzero amount, but not obviously all that much.
I continue to find the Carl Shulman vision alienating, a weird kind of middle ground and way of thinking and doing math. Is it convincing to some people, as a kind of existence proof? I have no idea.
Bill Gates predicts computer interfaces will become agent driven, but far more importantly that ASI is coming and there is no way to slow it down. He sees scaling as only having ‘two more cranks,’ video data and synthetic data, but expects success via improved metacognition that is more humanlike.
Andrej Karpathy talks at UC Berkeley, similarly predicts Her-style interface.
Q&A with Geoffrey Hinton.
Dario Amodei and Elad Gil talk to Google Cloud Next. Seemed inessential.
Oh Anthropic
Some troubling news.
At least one person in position to know has said no such agreement was ever offered to them, so this was at least not universal. We do not know how common it has been.
Open Weights Are Unsafe and Nothing Can Fix This
This came up during a Transfromer interview with Lawrence Lessig. Lessig is a strong advocate for open source in other contexts, but notices AI is different.
Rhetorical Innovation
A line attempted to be drawn to scale, Yudkowsky via Cameron of Dank EA Memes.
Remember that both sides of the line go out into the distance a very long way.
Aligning a Smarter Than Human Intelligence is Difficult
OpenAI offers a paper on using GPT-4 to find GPT-4’s mistakes.
They train the model to spot mistakes in code. It finds mistakes more efficiently than untrained GPT-4 and better than human evaluators. For now, a human-LLM combined team does better still by reducing false positives.
They partly used intentional tampering to introduce subtle bugs.
In addition to RLHF they use a technique called Force Sampling Beam Search (FSBS).
The critic also performed ‘out of sample’ on non-code examples, where it often managed to spot issues with samples previously rated by humans as flawless, sufficiently important that the humans no longer considered the samples flawless.
The conclusion is worth quoting:
Jan Leike, who contributed to this paper while still at OpenAI, offers thoughts here.
As a practical matter this all seems neat and helpful. The average accuracy of the evaluations will go up relative to human evaluations.
Code is easy mode, since the answer of whether it works is relatively objective. Value here is not so fragile. It is a good place to start. It also masks the dangers.
My concern is that this creates great temptation to rely on AI evaluations of AI, and to iterate repeatedly on those evaluations. It risks enshrining systematic correlated error, and amplifying those issues over time as the process feeds back upon itself. There are any number of ways that can go horribly wrong, starting with supercharged versions of all the usual Goodhart’s Law problems.
The average scoring, including the average human spot check, will look good for as long as we can understand what is going on, if we execute on this reasonably. Performance will genuinely be better at first. That will add to the temptation. Then the results will increasingly diverge.
Here is another example of going down a similar path.
I do like the idea of working with preference probabilities. I worry about working self-play into the picture, as it seems likely to exacerbate our Goodhart’s Law issues.
People Are Worried About AI Killing Everyone
A wrong but useful model of AI risk is attempted.
Several distinct things are usefully wrong here.
A few thoughts.
Our estimate of the path of future AI existential risk over time is changing like any good Bayesian estimate. Some events or information make the risk go up, some make it go down. Some insights make our estimate go up or down by revealing what was already true, others represent choices made by people.
Eventually, yes, the risk in the short term (~1 year or less let’s say), either of the event happening or us passing a ‘point of no return’ where in practice we are incapable of responding, starts to go up. From an outside view that may look steady, from an inside view it probably involves one or more large step changes as well, on key private and public decisions or on passage of time to critical points.
Top ten obvious examples after five minutes of thinking:
Some of these could be gradual, but many are likely or inherently sudden.
In particular, tie in the ability to take control versus the risk of it happening.
The traditional Yudkowsky or sharp left turn scenario is that these are the same thing. The highly intelligent and capable AI is going to attempt to take control if and only if it is confident that attempt would succeed at letting it fulfill its objectives (or it might well work and the risks of waiting are greater). The logic is obvious, and humans do their best to follow that logic as well.
Then there is the idea of a battle between ‘an AI adversary’ and ‘the human side.’
Why is this conflict not gamed out more?
Because there are mostly two groups of people here.
When someone in group #1 talks to someone in group #2, the goal is to convince people to accept the obvious. So you don’t game out exactly how the conflict works in practice or what the threshold is. You instead see what their absurd excuse or hopium is, and shoot it down and overwhelm it, and then they adjust, and you do it again. Occasionally this works and they become enlightened. When that happens, you are happy, great talk, but you are not closer to figuring out where the thresholds are.
When people in group #1 talk to each other about this, they still have radically different assumptions about among other things which AIs are against you and threat vectors and what scenarios might look like and how various things would work or people would react, and also the real scenarios involve effectively smarter things than you and also the details depend on unknown things about the future path of capabilities and conditions. So it is still super hard to make progress. And responding to a particular scenario on the margin based on how you think the battle would go is unlikely to turn losses into wins.
Mostly my answer is ‘yes, if capabilities do not stall we will effectively pass this point.’
Other People Are Not As Worried About AI Killing Everyone
From last week in audio: Aravind Srinivas, CEO of Perplexity, played a jarring mix of great founder and idiot disaster monkey on Lex Fridman. The parts where he describes the practical business of Perplexity are great, assuming he is not making things up. Then he will speculate about a future full of powerful AI agents doing everything, and say ‘I am not worried about AIs taking over’ as a throwaway line and get back to talking about other things, or say that open sourcing is the way to go because most people won’t have enough compute to do anything dangerous with the models.
I suspect that when Aravind says not worried, he and many others mean that literally.
As in, what me worry?
Or as in the way most people find a way to not worry about death.
It is not that Aravind thinks this will not happen. We all know that the planetary death rate is holding steady at 100%, but what is the point of going all existential angst about it? If AI is likely to get us all killed somewhat faster this round, well, that’s unfortunate but in the meantime let’s go build one of those great companies and worry about it later.
He then combines this with failure to feel the AGI. He is super excited for exactly the AIs that he expects, which will be able to be creative like Einstein, do tons of that thinking without humans present and come back to you, act as your agents, and do all the other cool things, exactly enough to be maximally awesome for humans, but not so much that humans have to worry about loss of control.
How is that possible? Is there even a narrow window of theoretical capability where you can have those abilities without the dangers? I mean, no, obviously there isn’t, but you can sort of pretend that there is and then also assume we will stabilize in exactly that part of the curve despite then discovering all of physics and so on.
The good news is that running Perplexity is almost entirely about being a great founder, so in practice what he does is mostly good. The ‘answer engine’ idea is great, and occasionally I find it the right tool for the right job although mostly I end up at either the Google Search or Claude Sonnet ends of the spectrum.
I do appreciate that ‘I don’t believe in ASI’ has moved from implied but unnoticed subtext to very clear text.
The Lighter Side
The longest kiss.
One missing word makes all the difference.