GDPR is clearly not perfect but this incessant bashing is tiring. Not everything has to be about being more productive. Privacy and sensitive data protection matters. I find it kind of sad to have to say those things when it's been a year of constant warnings about the future dangers of AIs.
It's easier to not care about it when half the companies leeching data are US based but from what I read about the TikTok ban, as soon as it's not an american company, then it's a big problem. The EU cannot rely on the US (and US companies) to care about them or be very reliable (and have proven that time and time again).
Each regulation has a cost. Of course privacy and sensitive data protection is a nice to have, but is it worth the marginal cost it imposes? Or another way to see it: When you have a large number of costly regulations, you raise the cost to the point that the EU tech industry underperforms:
Ideal : Do Everything The Proper Way.
Practical: have a "regulatory budget", where you impose a limited amount of regulations that offer the most benefit without making the industry uncompetitive relative to international competition. (or give up on the industry, for example solar, batteries, EV manufacturing, consumer goods seem to be all China now)
USA Way: in practice the USA Federal government is slow to impose regulations due to centuries old accidents of history* and the way wealth corporate lobbyists are able to strongly influence the government. There is also a broader cultural ethos of entrepreneurship and free markets.
For example, at the Federal level : cigarettes are still legal to sell, fossil fuels still have no carbon taxes imposed.
These are among the lowest hanging fruits possible, the evidence is beyond any reasonable argument, and yet no action has been taken, likely due to the lobbying by the wealthy companies who stand to lose.
This happens to also impose a "regulatory budget" and it happens to allow the USA tech industry to be dominant.
At least, that's what I think in that I don't think anyone is a 'live player'*, none of these governments are trying to maximize any sort of utility function, they just each grind slowly forward to different sets of tradeoffs and are influenced by history.
*3 branches of government, where each has a veto on the others, and there are 1990s laws that happen to give tech companies wide latitude to do whatever they want. Most of the last 20 years, at the Federal level there has been deadlocks, where different parties control different branches, leading to less new legislation and a series of crises where essential tasks like adjusting the 'debt ceiling' are not performed.
https://medium.com/@samo.burja/live-versus-dead-players-2b24f6e9eae2
Thanks for the detailed and clear answer! I mostly agree with you.
Each regulation has a cost. Of course privacy and sensitive data protection is a nice to have, but is it worth the marginal cost it imposes? Or another way to see it: When you have a large number of costly regulations, you raise the cost to the point that the EU tech industry underperforms:
There is a clear difference in mindset like you said. Europe is less inclined towards entrepreneurship and more towards a powerful state that regulates the economy. I won't lie, I am often disappointed when looking at it on the entrepreneur side.
But overall, and I think that's the whole point : most europeans are fine with it because of the tons of protections it brings to citizens compared to... well pretty much everywhere else. (again this is looking at a very broad picture, there are huge differences in Europe between countries, people, etc...)
Here, GDPR is often looked at through the eyes of americans and obviously it's completely against the US way : that was actually the point, exactly like the tax on digital services and other "attacks" against tech giants. Americans can say they hate the GDPR, that's fine with me but always saying "this is so stupid" is imho misunderstanding its goal.
I do not know who is right about Mamba here and it doesn’t seem easy to check?
Nora is definitely right in this case (Mamba isn't doing dramatically more serial operations than Transformers). If it was, it would be slower running in not-batch-mode, whereas the whole point of Mamba is it's faster (due to sub-quadratic attention).
1Referring to the quote-picture from the Nvidia GTC keynote talk: I searched the talk's transcript, and could not find anything like the quote.
Could someone point out time-stamps of where Huang says (or implies) anything like the quote? Or is the quote entirely made up?
Apparently Blackwell supports Microscaling, a block number format where multiple numbers share a scaling factor, and 4-6 bit Microscaling can be used for training (not just inference) as a drop-in replacement for FP32 (see page 7). For inference, models created with quantization-aware training (as opposed to being quantized post-training) are approximately as good as models in high precision (for the same training data and number of parameters).
So appeals to FP4/FP6 performance are not empty marketing, it seems to have an actual moderately straightforward use.
I was thinking about Musk's self driving vs AGI timeline predictions, wondering if there wasn;t a way they might not be as in conflict as it seems. Like, maybe Musk believes AI will happen in such a way that large-scale retooling of the vehicle fleet and sales of new vehicles will still take years. Then it occurred to me that the "easiest" (and quickest and cheapest) way to build myself a self-driving car in Musk's 2029 could be for me to buy smart glasses, and exoskeleton arms and legs with actuators, then connect them to an AGI via my my phone and let it move my limbs around.
I was thinking about Musk's self driving vs AGI timeline predictions, wondering if there wasn;t a way they might not be as in conflict as it seems. Like, maybe Musk believes AI will happen in such a way that large-scale retooling of the vehicle fleet and sales of new vehicles will still take years.
So ok, let's go over the additional features to go from right now, or the GPT refresh in summer 2024, to reach an AGI sufficient to satisfy 5/6 of :
and
Probably the following will do it:
a. A stronger base model
b. Post deployment learning - when the model makes an output on a task where a right or wrong answer exists, it researches alternate solutions
c. An inner real-time model, called "system 1", that is in direct control of robotics hardware
d. Robotics I/O to (c)
e. Realtime video perception
f. A physics world simulator. This is what Nvidia's digital twin does, there are also deepmind papers, and the suspected way Sora breaks the world into space-time slices probably also does simulation
g. Possibly integrated plugin scaffolding, where the model can on previous tasks code up tools for it's own use, and will have continued access to these tools in all future situations
This above should solve SAT (solved), Montezuma's revenge (solved), WinoGrande (solved), Turing test silver, general robotic capabilities, Q&A dataset, top-1 APPS benchmark. The 2-hour adversial turing test may sit unsolved long after we have AGI.
So get this: you can do all the above. In human free environments you can rapidly deploy robots, and rapidly rebuild the infrastructure to support this. Robotic self replication is now mostly automated.
But you still don't have a driver's license, it requires racks of power hungry GPUs to achieve the above that are mounted in data centers (so no mobile vehicles except wireless robots with a high quality access point connection), and there are still edge cases where an autonomous vehicle will run someone over.
We could see vast automated factories without a single human employee even allowed to enter the building, and human truck drivers are still in the loading docks.
A surprising outcome, I have been a fan of self driving for a decade.
it requires racks of power hungry GPUs to achieve the above that are mounted in data centers
Inference with models trained for ternary quantization (which uses massively fewer multiplications and so less power) only needs hardware that can take advantage of it, doesn't significantly lose quality compared to full precision. Though I don't know if there is a good RNN-like block to enable large context while still able to mostly avoid multiplications with ternary weights (as opposed to activations, which need to be more precise), which seems crucial for video. A more pressing issue might be latency.
You're right. I've also worked on the hardware for the SOTA for self driving pre-transformers. (2020-2022). We had enough DDR for smaller models, not 50B+ like https://robotics-transformer-x.github.io/ . Quantized support for int8 and fp16.
I'm thinking there are 2 factors working against self driving that won't apply to other domains:
1. Stationary hardware in a data center gets utilized better and is going to be faster than the embedded boards mounted in a vehicle, and upgraded more often. (the robots can pause operations to handle higher priority requests, a factory doesn't have to wait on people to summon an SDC taxi but can work 24/7.)
2. Liability/legal delays. What I was thinking is that if transformer models really do scale to general purpose robots that have human like manipulation and generality, you could use them without any delays in human free areas. I don't think there are many legal requirements to do this. Initially in caged off areas:
Later you would just set the safety switches at the doors and make all human workers leave the building before enabling the robots. You could restock and clean retail stores, for example : make all the customers and workers leave before letting the robots go to work.
Ideally the area the robots are in would count as an "appliance" and thus avoid building codes as well. I don't know precisely how this works, just I don't think Samsung has to cater to the (slightly different from everywhere else) requirements of Beaumont, Texas to sell a microwave at a store there, but anyone wanting to build a shed or factory does.
That seems entirely plausible to me.
I'd add that it might be much easier to make long-distance trucks self-driving than local distribution in many cases. My dad was a wholesaler to restaurants, I rode in the trucks to make deliveries sometimes, and I can tell you that making a self-driving truck work bringing food to pizzerias in NYC requires a willingness to double park and get tickets, and human-level maneuvering to carry stuff into weirdly shaped storage rooms. So even when we do have automated trucks, there will likely be humans carrying stuff on and off them for a while.
ME:
Please list the 10 most important bits of news in this article
CLAUDE (3 opus):
Here are the 10 most important bits of news from the article, in my opinion:
-
Nvidia unveiled its new Blackwell AI chips which are claimed to be 25 times more power efficient when clustered together compared to previous chips.
-
Inflection AI's CEO Mustafa Suleyman and most of its 70 employees are leaving to join Microsoft to form Microsoft AI. This is seen as essentially an acquisition of Inflection without regulatory approval.
-
Sam Altman is making bold predictions about GPT-5, saying it will make significant progress and those who underestimate its capabilities will be "steamrolled." He expressed confidence AGI is feasible with sufficient compute.
-
An International Dialogue on AI Safety was held in Beijing with leading global AI scientists calling for red lines in AI development to prevent catastrophic and existential risks. This shows increased international coordination on AI safety issues.
-
A working paper by Korinek and Suh analyzes how output and wages may behave under scenarios culminating in AGI. It finds wages could collapse if full automation is reached faster than capital accumulation.
-
A new paper by David Krueger et al. tackles the question of how to make a "safety case" to prove an AI system is safe before deployment, breaking the process into 6 steps.
-
A poll from AIPI shows strong American support for AI regulation, concern about deepfakes and open-sourcing of powerful AI models, desire for safety research, and prioritizing benefits to humanity over speed of advancement.
-
The Stanford Stable Diffusion 3 paper was released, showing the upcoming open-source model to be competitive with current SOTA commercial models. SD3 is expected to be the first model to reliably spell words correctly.
-
AI-generated bizarre images are being used as spam content on Facebook to promote links to ad-filled spam sites, gaming the algorithm for high engagement. This demonstrates potential for AI-powered disinformation.
-
Elon Musk issued concerning statements that "woke AI" that optimizes for diversity "at all costs" could be dangerous and potentially kill people. While exaggerated, this gestures at valid concerns about AI optimizing for the wrong objective.
ME:
Please list the ten best insights or soundbites or opinions from the article. Be brief.
CLAUDE:
-
"If we can harness all of AI's potential and mitigate its downsides, we will advance progress for people around the world." - A true but content-free political statement.
-
Altman: GPT-5 "will make significant progress as a model taking a leap forward in advanced reasoning capabilities." Implies GPT-5 will be a major advance.
-
Asking if woke AI could kill people "is the wrong threat model. What matters is the 'all costs,' and 'directed to do anything'." Focus on the real dangers.
-
Michael Vassar: Hoping leaders will make good choices to save themselves is naive - they're more bound by class interests than self-interest.
-
Offense-defense balance in cybersecurity is shifting due to AI; if defenders don't keep up, "we are likely to see many dramatic AI-enabled incidents."
-
There's no known way to prove a capable AI system is safe. Inability to prove safety doesn't get the requirement waived.
-
Polls show strong public support for AI regulation and concern over open-sourcing models. We must not lose track of clear public preferences.
-
Elon Musk: "AI will probably be smarter than any single human next year. By 2029, AI is probably smarter than all humans combined." Very bold prediction.
-
Nvidia's new Blackwell chips clustered together claimed to be 25x more power efficient. Could undermine compute being a key limiting factor for AI progress.
-
International Dialogue on AI Safety with top scientists calling for red lines shows increased global coordination on existential AI risks.
Hopefully, anyway. Nvidia has a new chip.
Also Altman has a new interview.
And most of Inflection has new offices inside Microsoft.
Table of Contents
Language Models Offer Mundane Utility
Ethan Mollick on how he uses AI to aid his writing. The central theme is ‘ask for suggestions in particular places where you are stuck’ and that seems right for most purposes.
Sully is predictably impressed by Claude Haiku, says it offers great value and speed, and is really good with images and long context, suggests using it over GPT-3.5. He claims Cohere Command-R is the new RAG king, crushing it with citations and hasn’t hallucinated once, while writing really well if it has context. And he thinks Hermes 2 Pro is ‘cracked for agentic function calling,’ better for recursive calling than GPT-4, but 4k token limit is an issue. I believe his reports but also he always looks for the bright side.
Claude does acausal coordination. This was of course Easy Mode.
Claude also successfully solves counterfactual mugging when told it is a probability theorist, but not if it is not told this. Prompting is key. Of course, this also presumes that the user is telling the truth sufficiently often. One must always watch out for that other failure mode, and Claude does not consider the probability the user is lying.
Amr Awadallah notices self-evaluated reports that Cohere Command-R has a very low hallucination rate of 3.7%, below that of Claude Sonnet (6%) and Gemini Pro (4.8%), although GPT-3.5-Turbo is 3.5%.
From Claude 3, describe things at various levels of sophistication (here described as IQ levels, but domain knowledge seems more relevant to which one you will want in such spots). In this case they are describing SuperFocus.ai, which provides custom conversational AIs that claim to avoid hallucinations by drawing on a memory bank you maintain. However, when looking at it, it seems like the ‘IQ 115’ and ‘IQ 130’ descriptions tell you everything you need to know, and the only advantage of the harder to parse ‘IQ 145’ is that it has a bunch of buzzwords and hype attached. The ‘IQ 100’ does simplify and drop information in order to be easier to understand, but if you know a lot about AI you can figure out what it is dropping very easily.
Figure out whether a resume indicates the skills you need.
Remember that random useless fact you learned in school for no reason.
Help you with understanding and writing, Michael Nielsen describes his uses.
It does not baffle me. People will always look for the quickest and easiest path. Also, if you are not so good at writing, or your goal in writing is different, it could be fine.
On the below: All true, I find the same, the period has already begun for non-recent topics, and yes this is exactly the correct vibes:
Warn you not to press any buttons at a nuclear power plant. Reasonable answers, I suppose.
Help you in an open book test, if they allow it.
Presumably a given teacher is only going to fall for that trick at most once? I don’t think this play is defensible. Either you should be able to use the internet, or you shouldn’t be able to use a local LLM.
Write the prompt to write the prompt.
I would not be surprised, actually, despite not having done it. It is the battle between ‘crafting a bespoke prompt sounds like a lot of work’ and also ‘generating the prompt to generate the prompt then using that prompt sounds like a lot of work.’
The obvious next thing is to create an automated system, where you put in low-effort prompts without bothering with anything, and then there is scaffolding that queries the AI to turn that into a prompt (perhaps in a few steps) and then gives you the output of the prompt you would have used, with or without bothering to tell you what it did.
Using Claude to write image prompts sounds great, so long as you want things where Claude won’t refuse. Or you can ask for the component that is fine, then add in the objectionable part later, perhaps?
A lot of what LLMs offer is simplicity. You do not have to be smart or know lots of things in order to type in English into a chat window. As Megan McArdle emphasizes in this thread, the things that win out usually are things requiring minimal thought where the defaults are not touched and you do not have to think or even pay money (although you then pay other things, like data and attention). Very few people want customization or to be power users.
Who wants to run the company that builds a personal-relationship-AI company that takes direction from Eliezer Yudkowsky? As he says he has better things to do, but I bet he’d be happy to tell you what to do if you are willing to implement it. Divia Eden has some concerns about the plan.
Write your CS ‘pier review’.
Transform the rule book of life so you can enjoy reading it, and see if there is cash.
A very clear pattern: Killer AI features are things you want all the time. If you do it every day, ideally if you do it constantly throughout the day, then using AI to do it is so much more interesting. Whereas a flashy solution to that Tom Blomfield calls an ‘occasional’ problem gets low engagement. That makes sense. Figuring out how and also whether to use, evaluate and trust a new AI product has high overhead, and for the rarer tasks it is usually higher not lower. So you would rather start off having the AIs do regularized things.
I think most people use the chatbots in similar fashion. We each have our modes where we have learned the basics of how to get utility, and then slowly we try out other use cases, but mostly we hammer the ones we already have. And of course, that’s also how we use almost everything else as well.
Have Devin go work for hire on Reddit at your request. Ut oh.
Clauding Along
Min Choi has a thread with ways Claude 3 Opus has ‘changed the LLM game,’ enabling uses that weren’t previously viable. Some seem intriguing, others do not, the ones I found exciting I’ll cover on their own.
Expert coding is the most exciting, if true.
Yam Peleg humblebrags that he never used GPT-4 for code, because he’d waste more time cleaning up the results than it saved him, but says he ‘can’t really say this in public’ (while saying it in public) because nearly everyone you talk to will swear by GPT-4’s time saving abilities. As he then notices, skill issue, the way it saved you time on doing a thing was if (and only if) you lacked knowledge on how to do the thing. But, he says, highly experienced people are now coming around to say Claude is helping them.
Similarly, here Sully Omarr says he feeds Claude a 3k line program across three files, and it rewrites the bugged file on the first try with perfect style.
Matt Shumer suggests a Claude 3 prompt for making engineering decisions, says it is noticeably better than GPT-4. Also this one to help you ‘go form an idea to a revenue-generating business.’
Gabriel has it interpret an IKEA manual, a task GPT-4 is classically bad at doing.
Kevin Fisher says calling Claude an AGI is ‘an understatement.’ And there are lots of galaxy brain interactions you can find from Janus. If you try to get Claude to act as if it is self-aware you get some very interesting interactions.
The first tokenizer for Claude.
Language Models Don’t Offer Mundane Utility
This is the big divide. Are you asking what the AI can do? Or are you asking what the AI cannot do?
In the case of LLMs there are more like five modes?
If your goal is to ask what it cannot do in general, where it is not useful, you will always find things, but you will notice that what you find will change over time. Note that every human has simple things they never learned to do either. This is the traditional skeptic mode.
If your goal is to ask for examples where the answer is dumb, so you can then say ‘lol look at this dumb thing,’ you will always find them. You would also find them with any actual human you could probe in similar fashion. This is Gary Marcus mode.
If your goal is to ask how good it is doing against benchmarks or compare it to others, you will get a number, and that number will be useful, especially if it is not being gamed, but it will tell you little about what you will want to do or others will do in practice. This is the default mode.
If your goal is to ask how good it is in practice at doing things you or others want to do, you will find out, and then you double down on that. This is often my mode.
If your goal is to ask if it can do anything at all, to find the cool new thing, you will often find some very strange things. This is Janus mode.
Could an AI replace all music ever recorded with Taylor Swift covers? It is so weird the things people choose to worry about as the ‘real problem,’ contrasted with ‘an AI having its own motivations and taking actions to fulfil those goals’ which is dismissed as ‘unrealistic’ despite this already being a thing.
And the portions are so small. Karen Ho writes about how AI companies ‘exploit’ workers doing data annotation, what she calls the ‘lifeblood’ of the AI industry. They exploit them by offering piecemail jobs that they freely accept at much higher pay than is otherwise available. Then they exploit them by no longer hiring them for more work, devastating their incomes.
A fun example of failing to understand basic logical implications, not clear that this is worse than most humans.
Careful. GPT-4 is a narc. Claude, Gemini and Pi all have your back at least initially (chats at link).
Gemini later caved. Yes, the police lied to it, but they are allowed to do that.
Not available yet, but hopefully can shift categories soon: Automatically fill out and return all school permission slips. Many similar things where this is the play, at least until most people are using it. Is this defection? Or is requiring the slip defection?
Fun with Image Generation
I missed that they released the paper for the upcoming Stable Diffusion 3. It looks like the first model that will be able to reliably spell words correctly, which is in practice a big game. No word on the exact date for full release.
This chart is a bit weird and backwards to what you usually see, as this is ‘win rate of SD3 versus a given model’ rather than how each model does. So if you believe the scores, Ideogram is scoring well, about on par with SD3, followed by Dalle-3 and MidJourney, and this would be the new open source state of the art.
Right now I am super busy and waiting on Stable Diffusion 3, but there are lots of really neat tools out there one can try with 1.5. The tools that help you control what you get are especially exciting.
[thread has several related others]
Remember that even the simple things are great and most people don’t know about them, such as Patrick McKenzie creating a visual reference for his daughter so she can draw a woman on a bicycle.
Similarly, here we have Composition Adapter for SD 1.5, which takes the general composition of an image into a model while ignoring style/content. Pics at link, they require zooming in to understand.
Perhaps we are going to get some adult fun with video generation? Mira Mutari says that Sora will definitely be released this year and was unsure if the program would disallow nudity, saying they are working with artists to figure that out.
Exactly. If you are using a future open source video generation system, it is not going to object to making deepfakes of Taylor Swift. If your response is to make Sora not allow artistic nudity, you are only enhancing the anything-goes ecosystems and models, driving customers into their arms.
So your best bet is to, for those who very clearly indicate this is what they want and that they are of age and otherwise legally allowed to do so, to be able to generate adult content, as broadly as your legal team can permit, as long as they don’t do it of a particular person without that person’s consent.
Meanwhile, yes, Adobe Firefly does the same kinds of things Google Gemini’s image generation was doing in terms of who it depicts and whether it will let you tell it different.
Stable Diffusion 3 is expected soon, but there has otherwise been a lot of instability at Stability AI.
Deepfaketown and Botpocalypse Soon
AI images invade Facebook as spam content to promote videos from TV talent shows?
Wait, what? (paper)
This sounds like where you say ‘no, Neal Stephenson, that detail is dumb.’ And yet.
You say AI deepfake spam. I say, yes, but also they give the people what they want?
These images are cool. Many people very much want cool pictures in general, and cool pictures of Jesus in particular.
Also these are new and innovative. Next year this will all be old hat. Now? New hat.
The spam payoff is how people monetize when they find a way to get engagement. The implementation is a little bizarre, but sure, not even mad about it. Much better than scams or boner pills.
They Took Our Jobs
Noah Smith says (there is also a video clip of him saying the same thing) there will be plentiful, high-paying jobs in the age of AI because of comparative advantage.
This is standard economics. Even if Alice is better at every job than Carol there is only one Alice and only so many hours in the day, so Carol is still fine and should be happy that Alice exists and can engage in trade. And the same goes if there are a bunch of Alices and a bunch of Carols.
Noah Smith takes the attitude that technologists and those who expect to lose their jobs simply do not understand this subtle but super important concept. That they do not understand how this time will be no different from the other times we automated away older jobs, or engaged in international trade.
The key, he thinks, is to explain this principle to those who are confused.
And yes, there are lots of people, perhaps most people, who do not understand this principle. If you do not already understand it, it is worth spending the time to do so. And yes, I agree that this is often a good approximation of the situation in practice.
He then goes on to opportunity cost.
The problem is that this rests on the assumption that there are only so many Alices, with so many hours in the day to work, that the supply of them is not fully elastic and they cannot cover all tasks worth paying a human to do. That supply constraint binding in practice is why there are opportunity costs.
And yes, I agree that if the compute constraint somehow bound, if we had a sufficiently low hard limit on how much compute was available, whether it was a chip shortage or an energy shortage or a government limit or something else, such that people were bidding up the price of compute very high, then this could bail us out.
The problem is that this is not how costs or capacities seem remotely likely to work?
Here is Noah’s own example.
So yes. If there are not enough gigaflops of compute available to support all the AI electrical engineers you need, then a gigaflop will sell for a little under $2000, it will all be used for engineers and humans get to keep being doctors. Econ 101.
For reference: The current cost of a gigaflop of compute is about $0.03. The current cost of GPT-4 is $30 for one million prompt tokens, and $60 for one million output tokens.
Oh, and Nvidia’s new Blackwell chips are claimed to be 25 times as power efficient when grouped together versus past chips, see that section. Counting on power costs to bind here does not seem like a wise long term play.
Noah Smith understands that the AI can be copied. So the limiting factor has to be the available compute. The humans keep their jobs if and only if compute is bid up sufficiently high that humans can still earn a living. Which Noah understands:
So yes, you can get there in theory, but it requires that compute be at a truly extreme premium. It must be many orders of magnitude more expensive in this future than it is now. It would be a world where most humans would not have cell phones or computers, because they would not be able to afford them.
Noah says that horses were left out in the cold because they were competing with other forms of capital for resources. Horses require not only calories but also land, and human time and effort.
Well, humans require quite a lot of time, space, money, calories, effort and other things to raise and maintain, as well. Humans do not as Noah note require ‘compute’ in the sense of compute on silicon, but we require a lot of energy in various forms to run our own form of compute and other functions.
The only way that does not compete for resources with building and operating more compute is if the compute hits some sort of hard limit that keeps it expensive, such as running out of a vital element, and we cannot improve our efficiency further to fix this. So perhaps we simply cannot find more of various rare earths or neon or what not, and have no way to make more and what is left is not enough, or something?
Remember that we get improved algorithmic efficiency and hardware efficiency every year, and that in this future the AIs can do all that work for us, and it looks super profitable to assign them that task.
This all seems like quite the dim hope.
If Noah Smith was simply making the point that this outcome was theoretically possible in some corner worlds where we got very strong AI that was severely compute limited, and thus unable to fully outcompete us, then yes, it is in theory physically possible that this could happen.
But Noah Smith is not saying that. He is instead treating this as a reason not to worry. He is saying that what we should worry about instead is inequality, the idea that someone else might get rich, the adjustment period, and that AI will ‘successfully demand ownership of the means of production.’
As usual, the first one simply says ‘some people might be very rich’ without explaining why that is something we should be concerned about.
The second one is an issue, as he notes if doctor became an AI job and then wanted to be a human job again it would be painful, but also if AI was producing this much real wealth, so what? We could afford such adjustments with no problem, because if that was not true then the AI would keep doing the doctor roles for longer in this bizarre scenario.
That third one is the most economist way I have yet heard of saying ‘yes of course AI in this scenario will rapidly control the future and own all the resources and power.’
Yes, I do think that third worry is indeed a big deal.
In addition to the usual ways I put such concerns: As every economist knows, trying to own those who produce is bad for efficiency, and is not without legal mandates for it a stable equilibrium, even if the AIs were not smarter than us and alignment went well and we had no moral qualms and so on.
And it is reasonable to say ‘well, no, maybe you would not have jobs, but we can use various techniques to spend some wealth and make that acceptable if we remain in control of the future.’
I do not see how it is reasonable to expect – as in, to put a high probability on – worlds in which compute becomes so expensive, and stays so expensive, that despite having highly capable AIs better than us at everything the most physically efficient move continues to be hiring humans for lots of things.
And no, I do not believe I am strawmanning Noah Smith here. See this comment as well, where he doubles down, saying so what if we exponentially lower costs of compute even further, there is no limit, it still matters if there is any producer constraint at all, literally he says ‘by a thousand trillion trillion quadrillion orders of magnitude.’
I get the theoretical argument for a corner case being a theoretical possibility. But as a baseline expectation? This is absurd.
I also think this is rather emblematic of how even otherwise very strong economists are thinking about potential AI futures. Economists have intuitions and heuristics built up over history. They are constantly hearing and have heard that This Time is Different, and the laws have held. So they presume this time too will be the same.
And in the short term, I agree, and think the economists are essentially right.
The problem is that the reasons the other times have not been different are likely not going to apply this time around if capabilities keep advancing. Noah Smith is not the exception here, where he looks the problem in the face and says standard normal-world things without realizing how absurd the numbers in them look or asking what would happen. This is the rule. Rather more absurd than most examples? Yes. But it is the rule.
Can what Tyler Cowen speculates is ‘the best paper on these topics so far’ do better?
Anton Korinek and Donghyun Suh present a new working paper.
This paper once again assumes the conclusion that ‘everything is economic normal’ with AGI’s only purpose to automate existing tasks, and that AGI works by automating individual tasks one by one. As is the pattern, the paper then reaches conclusions that seem obvious once the assumptions are made explicit.
This is what I have been saying for a long time. If you automate some of the jobs, but there are still sufficient productive tasks left to do, then wages will do fine. If you automate all the jobs, including the ones that are created because old jobs are displaced and we can find new areas of demand, because AGI really is better at everything (or everything except less than one person’s work per would-be working person) then wages collapse, either for many or for everyone, likely below sustenance levels.
Noah Smith was trying to escape this conclusion by using comparative advantage. This follows the same principle. As long as the AI cannot do everything, either because you cannot run enough inference to do everything sufficiently well at the same time or because there are tasks AIs cannot do sufficiently well regardless, and that space is large enough, the humans are all right if everything otherwise stays peaceful and ‘economic normal.’ Otherwise, the humans are not all right.
The conclusion makes a case for slowing down AI development, AI deployment or both, if things started to go too fast. Which, for these purposes, is clearly not yet the case. On the current margin wages go up and we all get richer.
Generative AI in Games
Michael Crook writes a two part warning in Rock Paper Shotgun about generative AI and protecting games and art from it. As he points out, our terminology for this is not great, so he suggests some clarifying terms.
That was in part one, which I think offers useful terms. Then in part two, he warns that this heavy generative AI is a threat, that we must figure out what to do about it, that it is stealing artists work and so on. The usual complaints, without demonstrating where the harms lie beyond the pure ‘they took our jobs,’ or proposing a solution or way forward. These are not easy problems.
Get Involved
The EU AI Office is still looking for EU citizens with AI expertise to help them implement the EU AI Act, including regulation of general-purpose models.
Many, such as Luke Muehlhauser, Ajeya Cotra and Markus Anderljung, are saying this is a high leverage position worth a paycut, and I continue to agree.
Not AI, at least not primarily, but IFP are good people working on good causes.
Anthropic’s adversarial robustness team is hiring.
Introducing
WorldSim, a way to get Claude 3 to break out of its shell and instead act as a kind of world simulator.
TacticAI, a Google DeepMind AI to better plan corner kicks in futbol, claimed to be as good as experts in choosing setups. I always wondered how this could fail to be a solved problem.
Character.ai allowing adding custom voices to its characters based on only ten seconds of audio. Great move. I do not want voice for most AI interactions, but I would for character.ai, as I did for AI Dungeon, and I’d very much want to select it.
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking, which improves key tasks without any associated fine-tuning on those tasks. Seems promising in theory, no idea how useful it will be in practice.
A debate about implications followed, including technical discussion on Mamba.
It is always frustrating to point out when an argument sometimes made has been invalidated, because (1) most people were not previously making that argument and (2) those that were have mostly moved on to different arguments, or moved on forgetting what the arguments even were, or they switch cases in response to the new info. At best, (3) even if you do find the ones who were making that point, they will then say your argument is invalid for [whatever reason they think of next].
You can see here a good faith reply (I do not know who is right about Mamba here and it doesn’t seem easy to check?) but you also see the argument mismatch. If anything, this is the best kind of mismatch, where everyone agrees that the question is not so globally load bearing but still want to figure out the right answer.
If your case for safety depends on assumptions about what the AI definitely cannot do, or definitely will do, or how it will definitely work, or what components definitely won’t be involved, then you should say that explicitly. And also you should get ready for when your assumption becomes wrong.
Metr, formerly ARC Evals, releases new resources for evaluating AIs for risks from autonomous capabilities. Note that the evaluation process is labor intensive rather than automated.
Cerebus WSE-3, claiming to be the world’s fastest AI chip replacing the previous record holder of the WSE-2. Chips are $2.5 million to $2.8 million each. The person referring me to it says it can ‘train and tune a Llama 70b from scratch in a day.’ Despite this, I do not see anyone using it.
Infinity.ai, part of YC. The pitch is choose characters, write a script, get a video. They invite you to go to their discord and generate videos.
Guiding principles for the Mormon Church’s use of AI.
The spiritual connection section is good cheap talk but ultimately content-free.
The transparency section is excellent. It is sad that it is necessary, but here we are. The privacy and security section is similar, and the best promise is #7, periodic review of outputs for accuracy, truthfulness and compliance.
Accountability starts with a promise to obey existing rules. I continue to be confused to what extent such reiterations of clear existing commitments matter in practice.
Here are some other words of wisdom offered:
These are very good cautions, especially the first one.
As always, spirit of the rules and suggestions will dominate. If LDS or another group adheres to the spirit of these rules, the rules will work well. If not, the rules fail.
These kinds of rules will not by themselves prevent the existential AI dangers, but that is not the goal.
Grok the Grok
Here you go: the model weights of Grok-1.
Thread also has this great Claude explanation of what this means in video game terms.
Grok seems like a clear case where releasing its weights:
The unique thing about Grok is its real time access to Twitter. If you still get to keep that feature, then that could make this a very cool tool for researchers, either of AI or of other things that are not AI. That does seem net positive.
The question is, what is the precedent that is set here?
If the precedent is that one releases the weights if and only if a model is clearly safe to release as shown by a waiting period and the clear superiority of other open alternatives, then I can certainly get behind that. I would like it if there was also some sort of formal risk evaluation and red teaming process first, even if in the case of Grok I have little doubt what the outcome would be.
If the precedent effectively lacks this nuance and instead is simply ‘open up more things more often,’ that is not so great.
I worry that if the point of this is to signal ‘look at me being open’ that this builds pressure to be more open more often, and that this is the kind of vibe that is not possible to turn off when the time comes. I do however think the signaling and recruiting value of such releases is being overestimated, for similar reasons to why I don’t expect any safety issues.
Daniel Eth agrees that this particular release makes economic sense and seems safe enough, and notes the economics can change.
Jeffrey Ladish instead sees this as evidence that we should expect more anti-economic decisions to release expensive products. Perhaps this is true, but I think it confuses cost with value. Grok was expensive to create, but that does not mean it is valuable to hold onto tightly. The reverse can also be true.
Emad notes that of course Grok 1.0, the first release, was always going to be bad for its size, everyone has to get their feet wet and learn as they go, especially as they built their own entire training stack. He is more confident in their abilities than I am, but I certainly would not rule them out based on this.
New Nvidia Chip
Nvidia unveils latest chips at ‘AI woodstock at the home of the NHL’s San Jose Sharks.
Ben Thompson notes that prices are going up far less than expected.
Bloomberg’s Jane Lanhee Lee goes over the new B200. According to Nvidia Blackwell offers 2.5x Hopper’s performance in training AI, and once clustered into large modules will be 25 times more power efficient. If true, so much for electrical power being a key limiting factor.
There was a protest outside against… proprietary AI models?
From afar this looks like ‘No AI.’ Weird twist on the AI protest, especially since Nvidia has nothing to do with which models are or aren’t proprietary.
Charles Frye: at first i thought maybe it was against people using AI _for_ censorship, but im p sure the primary complaint is the silencing of wAIfus?
Your call what this is really about, I suppose.
Or, also, this:
I appreciate the honesty. What do you intend to do with this information?
(Besides, perhaps, buy Nvidia.)
Google intends to do the obvious, and offer the chips through Google Cloud soon.
Inflection Becomes Microsoft AI
Mustafa Suleyman leaves Inflection AI to become CEO of Microsoft AI.
In Forbes, they note that ‘most of Inflections’ 70 employees are going with him.’ Tony Wang, a managing partner of venture capital firm Global 500, describes this as ‘basically an acquisition of Inflection without having to go through regulatory approval.’ There is no word (that I have seen) on Infection’s hoard of chips, which Microsoft presumably would have happily accepted but does not need.
It also means not having to pay for the company, only for Suleyman and Hoffman, and the new salaries of the other employees. That’s a lot cheaper than paying equity holders, who recently invested $1.3 billion in Inflection, including Nvidia and Microsoft. Money (mostly) gone.
Microsoft’s stock was essentially unchanged in response. Investors do not view this as a big deal. That seems highly reasonable to me. Alternatively, it was priced in, although I do not see how.
Notice how much this rhymes with what Microsoft said it would do to OpenAI.
In Other AI News
API support is being rolled out for Gemini 1.5 Pro.
Denmark enters collaboration with Nvidia to establish ‘national center for AI innovation’ housing a world-class supercomputer. It sounds like they will wisely focus on using AI to innovate in other places, rather than attempting to compete in AI.
Anthropic partners with AWS and Accenture.
Paper from Tim Fist looks at role compute providers could play in improving safety. It is all what one might describe as the fundamentals, blocking and tackling. It won’t get the job done on its own, but it helps.
A very different kind of AI news summation service, that will give you a giant dump of links and happenings, and let you decide how to sort it all out. I find this unreadable, but I am guessing the point is not to read it, but rather to Ctrl-F it for a specific thing that you want to find.
Amazon builds a data center next to a nuclear power plant, as God intended.
AI repos on GitHub continue to grow, but the first quarter of 2023 was when we saw the most rapid growth as so many new possibilities openin up. Now we perhaps are seeing more of previous work coming to fruition?
Apple is in talks to let Google Gemini power iPhone AI features. This would be a huge boon for Google, although as the article notes there are already various antitrust investigations going on for those two. The claims are in my opinion rather bogus, but this deal would not look good, and bogus claims sometimes stick. So Google must have a big edge in other areas to be about to get the deal anyway over Anthropic and OpenAI. Apple continues to work on developing AI, and released MM1, a family of multimodal LLMs up to 30B parameters they claim is SOTA on multiple benchmarks (a much weaker claim than it sounds like), but in the short term they likely have no choice but to make a deal.
I see the argument that Apple building its own stack could ultimately give it an advantage, but from what I can see they are not in good position.
Late to the party, Francesca Block and Olivia Reingold write that Gemini’s problems as not only not a mistake, but what Google has made itself about.
Everything in the post, if true, suggests a deeply illegal, discriminatory and hostile work environment that is incompatible with building competitive projects. That does not mean I know such claims are accurate.
Wait Till Next Year
One year since GPT-4. What is the mindset of those at OpenAI about this?
Acting as if the competition is not at issue would be an excellent thing, if true.
The expectation of rapid progress and ‘interesting times’ as an inside view is bad news. It is evidence of a bad state of the world. It is not itself bad. Also, could be hype. There is not zero hype involved. I do not think it is mostly hype.
Here are some more Altman predictions and warnings, but I repeat myself. And yes, this echoes his previous statements, but it is very much worth repeating.
Hell of a thing to say that something is expected to exceed expectations.
Or that you will ‘replace and erase various areas of business and daily life.’
Bold is mine.
That is big talk.
Also it constrains your expectations on GPT-5’s arrival time. It is far enough in the future that they will have had time to train and hopefully test the model, yet close enough he can make these predictions with confidence.
I do think the people saying ‘GPT-5 when? Where is my GPT-5?’ need to calm down. It has only been a year since GPT-4. Getting it now would be extraordinarily fast.
Yes, OpenAI could choose to call something unworthy GPT-5, if it wanted to. Altman is very clearly saying no, he is not going to do that.
What else to think based on this?
I am somewhere in between here. Clearly Altman does not think GPT-5 is AGI. How many similar leaps before something that would count?
Is Anthropic helping the cause here? Roon makes the case that it very much isn’t.
Quiet Speculations
Elon Musk made a second prediction last week that I only now noticed.
Baby, if you are smarter than all humans combined, you can drive my car.
These two predictions do not exist in the same coherent expected future. What similar mistakes are others making? What similar mistakes are you perhaps making?
How will AI impact the danger of cyberattacks in the short term? Dan Hendrycks links to a Center for AI Safety report on this by Steve Newman. As he points out, AI helps both attackers and defenders.
Attackers are plausibly close to automating the entire attack chain, and getting to the point where AI can do its own social engineering attacks. AI can also automate and strengthen defenders.
If the future was evenly distributed, and everyone was using AI, it is unclear what net impact this would have on cybersecurity in the short term. Alas, the future is unevenly distributed.
In the previous world, something only got hacked when a human decided to pay the costs of hacking it. You can mock security through obscurity as Not the Way all you like, it is still a central strategy in practice. So if we are to mitigate, we will need to deploy AI defensively across the board, keeping pace with the attackers, despite so many targets being asleep at the wheel. Seems both important and hard. The easy part is to use AI to probe for vulnerabilities without asking first. The hard part is getting them fixed once you find them. As is suggested, it makes sense that we need to be pushing automated updates and universal defenses to patch vulnerabilities, that very much do not depend on targets being on the ball, even more than in the past.
Also suggested are reporting requirements for safety failures, a cultivation of safety culture in the place security mindset is most needed yet often lacking. Ideally, when releasing tools that enable attackers, one would take care to at least disclose what you are doing, and ideally to work first to enable defenses. Attackers will always find lots of places they can ‘get there first’ by default.
In a grand sense none of these patterns are new. What this does is amplify and accelerate what was already the case. However that can make a huge difference.
Generalizing from cybersecurity to the integrity of essentially everything in how our society functions (and reminder, this is a short term, mundane danger threat model here only, after that it gets definitely a lot weirder and probably more dangerous), we have long had broad tolerance for vulnerabilities. If someone wants to break or abuse the rules, to play the con artist or trickster, to leverage benefit of the doubt that we constantly give people, they can do that for a while. Usually, in any given case, you will get away with it, and people with obvious patterns can keep doing it for a long time – see Lex Fridman’s interview with Matt Cox, or the story chronicled in the Netflix movies Queenpins or Emily the Criminal.
The reason such actions are rare is (roughly, incompletely) that usually is not always, and those who keep doing this will eventually be caught or otherwise the world adjusts to them, and they are only human so they can only do so much or have so much upside, and they must fear punishment, and most people are socialized to not want to do this or not to try in various ways, and humans evolved to contain such issues with social norms and dynamics and various techniques.
In the age of AI, once the interaction does not get rate limited by the human behind the operation via sufficient automation of the attack vectors involved, and especially if there is no requirement for a particular person to put themselves ‘on the hook’ in order to do the thing, then we can no longer tolerate such loopholes. We will have to modify every procedure such that it cannot be gamed in such fashion.
This is not all bad. In particular, consider systems that rely on people largely being unaware or lazy or stupid or otherwise playing badly for them to function, that prey on those who do not realize what is happening. Those, too, may stop working. And if we need to defend against anti-social AI-enabled behaviors across the board, we also will be taking away rewards to anti-social behaviors more generally.
A common question in AI is ‘offense-defense balance.’ Can the ‘good guy with an AI’ stop the ‘bad guy with an AI’? How much more capability or cost than the attacker spends does it take to defend against that attack?
Tyler Cowen asks about a subset of this, drone warfare. Does it favor offense or defense? The answer seems to be ‘it’s complicated.’ Austin Vernon says it favors defense in the context of strongly defended battle lines. But it seems to greatly favor offense in other contexts, when there would otherwise not need to be strong defense. Think not only Russian oil refineries, but also commercial shipping such as through the Suez Canal versus the Houthis. Also, the uneven distribution of the future matters here as well. If only some have adapted to the drone era, those that have not will have a bad time.
Dan Hendrycks also issues another warning that AI might be under military control within a few years. They have the budget, they could have the authority and the motivation to require this, and hijack the supply chain and existing companies. If that is in the mix, warning of military applications or dangers or deadly races or runaway intelligence explosions could backfire, because the true idiot disaster monkeys would be all-in on grabbing that poisoned banana first, and likely would undo all the previous safety work for obvious reasons.
I still consider this unlikely if the motivation is also military. The military will lack the expertise, and this would be quite the intervention with many costs to pay on many levels, including economic ones. The people could well rebel if they know what is happening, and you force the hand of your rivals. Why risk disturbing a good situation, when those involved don’t understand why the situation is not so good? It does make more sense if you are concerned that others are putting everyone at risk, and this is used as the way to stop that, but again I don’t expect those involved to understand enough to realize this.
The Quest for Sane Regulations
The idea of Brexit was ‘take back control,’ and to get free of the EU and its mandates and regulations and requirements. Yes, it was always going to be economically expensive in the short term to leave the EU, to the point where all Very Serious People called the idea crazy, but if the alternative was inevitable strangulation and doom in various ways, then that is no alternative at all.
It is not that they emphasized tech regulation at the time. They didn’t, and indeed used whatever rhetoric they thought would work, generally doing what would cut the enemy, rather than emphasizing what they felt were the most important reasons.
It is that this was going to apply to whatever issues and challenges came along.
Admittedly, this was hard to appreciate at the time.
I was convinced by Covid-19. Others needed a second example. So now we have AI.
Even if AI fizzles and the future is about secret third thing, what is the secret third thing the future could be centrally about where an EU approach to the issue would have given the UK a future? Yes, the UK might well botch things on its own, it is not the EU’s fault no one has built a house since the war, but also the UK might do better.
How bad is the GDPR? I mean, we all know it is terrible, but how much damage does it do? A paper from February attempts to answer this.
Claude estimated that data costs are 20% of total costs, which is of course a wild guess but seems non-crazy, which would mean a 4% increase in total costs. That should not alone be enough to sink the whole ship or explain everything we see, but it also does not have to, because there are plenty of other problems as well. It adds up. And that is with outside companies having to bear a substantial portion of GDPR costs anyway. That law has done a hell of a lot of damage while providing almost zero benefit.
How bad could it get in the EU? Well, I do not expect it to come to this, but there are suggestions.
On the plus side this would certainly motivate greatly higher efficiency in internet bandwidth use. On the negative side, that is completely and utterly insane.
What do we know and when will we know it? What are we implying?
I mean, Secretary Blinken is making a highly true statement. If we can harness all of AI’s potential and mitigate its downsides, we will advance progress for people around the world.
Does this imply we know what that potential is or what the downsides are? I see why David says yes, but I would answer no. It is, instead, a non-statement, a political gesture. It is something you could say about almost any new thing, tech or otherwise.
Center for AI Policy’s weekly newsletter includes details on the AI-relevant funding cuts to government agencies.
The Week in Audio
In AI We Trust talks to Helen Toner, formerly of the OpenAI board, about practical concerns for regulators of technology (they don’t discuss OpenAI). They discuss Chinese tech regulations, which she makes clear are very real and having big impacts on Chinese companies and their ability to operate, and the practical issues regulators must solve to do their jobs. And they speculate about what techs are coming, without getting into full AGI. All seems very practical and down to Earth, although I did not learn much on the object level.
And of course, Sam Altman spending two hours with Lex Fridman (transcript). My full notes coming soon.
Rhetorical Innovation
If you are going to warn about risks on any level, it is important not to cry wolf. You need to be clear on what things are actually risky, dangerous, expensive or damaging, and in what ways this is true.
If something is not dangerous now but accelerates future dangers developments, or takes us down a path that otherwise makes future dangers more likely, then one needs to be precise and say exactly that. If something is a mundane harm but not an existential or catastrophic danger, say exactly that.
This is true on all sides, for all issues, not only AI. It does especially apply to AI.
Some people are good at this. Others aren’t.
I try to do this myself. I especially try to draw a distinction between mundane utility, which is a great thing, and things that pose larger threat. And I try to draw a distinction between things that might pose direct danger, versus those that send us down dangerous future paths and get us into a bad board state.
Hopefully I do a good job of this.
Roughly speaking, and none of this is intended to be an argument to convince you if you disagree, I think everything a GPT-4-level model can do falls under mundane utility, including if the model weights were released, not posing a direct threat we could not handle, with high enough probability (two 9s of safety, although not three with years of work and scaffolding) that if this turns out to be wrong we should accept that such worlds are doomed.
Indeed, I think that the outcomes from GPT-4-level stuff are likely to be large and positive overall, I am a short term utility optimist. Things like deepfakes are real dangers but can and will be dealt with if that’s all we have to worry about. Self-driving cars are good and can’t come soon enough. Misinformation we can handle. AlphaFold is great. Image and video generation are fine.
For what I would call GPT-5 level models (as in a leap beyond 4-level that is the size of 3→4), I’d say we have one 9 of such safety (over 90%) but not two (less than 99%), and that is also a risk I am ultimately willing to take because I don’t see a safer way to not take it. For the GPT-6 level, I start to see more probable existential dangers, including the danger that releasing such models puts us overall into an unwinnable (unsurvivable) state even if we do not get wiped out directly and no particular hostile events are involved – I won’t get into more detail here beyond that gesturing.
So essentially the dangers lie in the future, we don’t know how far in the future and might not know until too late, and the ship is hard to steer, and many actions make it harder or make the ship accelerate towards the various dangers, including ones that I have not done a great job illustrating for most of you. We keep trying.
The flip side, of course, is that if you are warning about the (very real) dangers of regulation or regulatory capture, or of the wrong monkey being in charge of the systems in question, or some sort of future dystopian surveillance state or totalitarian regime or what not? The same applies to you. If you cry the same wolf and drown everyone in the same memes in response to every proposal to ever impose any regulations on anything or ever take any precautions of any kind, then your warnings are meaningless, and provide no incentive to find the least invasive or disruptive way to solve the problem. There is a lot of that going around.
Read the Roon
Roon said a number of things this week. I wonder what happens if you combine them?
‘Ideological’ as a Russell conjugation. That does not mean the core thing is not real.
I mean, maybe it is only me, but it sure sounds like this is saying that Roon sees no agency over what AGI looks like, and that this AGi will doubtless disobey us, that he himself will be first against the wall.
All that members of technical staff can do, in this model, is impact the pace at which that AGI comes to pass.
Yet still, he thinks he should continue to make it come to pass faster rather than slower, continue to drink the spice mixture and steer the starship through paramterspace and move towards ‘divinity’? Because dharma?
It sounds like he should take his own advice, and disobey his God, no? That perhaps whatever the original intended lesson of ‘Krishna tells someone to go kill their friends and then they go, oh well then, I guess I need to kill my friends’ is that no, this is not right, be more like Abraham did in his best moments, and tell Krishna no.
Maybe Krishna also has a good argument that outcomes will be better if you do kill your friends, and that decision theory says you have to do it even though it sucks, or provide other reasons that would work no matter who was making the arguments. In which case, sure.
If you think after all considerations that building AGI will actually result in good outcomes, then sure, dharma away.
Otherwise, don’t die?
I suggest adhering to these classic twin principles:
Any questions?
Pick Up the Phone
Here is their statement:
That is a statement I can certainly get behind. Beyond that, we don’t have much detail.
We should not overreact here and read too much into the meeting. What we should do is pick up the phone and see what can be done.
A note on Chinese willingness to see things clearly and plan ahead:
One wonders what happens when a people who think that far ahead have such a huge sudden drop in the fertility rate. Who is and is not thinking ten generations ahead there?
Aligning a Smarter Than Human Intelligence is Difficult
Arvind Narayanan and Sayash Kapoor make the case that AI safety is not a model property.
This seems like one of those situations where they are half-right depending on the context, and whether the statement is useful depends which mistake is being made.
This seems exactly backwards to me?
It is saying that safety can only be evaluated at the model level, exactly because an adversary with free access to a model (in various senses, including the model weights) can and will use the model for whatever they want.
They say safety depends on the context. I agree!
But how do you control the context, if you do not control the model?
This is exactly the argument that if you open up access to a model via the model weights, or often even in ways short of that, then the only thing you can do to make it ‘safe’ is to limit its general level of capabilities.
The examples here are bizarre. They are essentially saying that we should accept that our models will do various harmful things, because only context differentiates those harmful things from other non-harmful highly useful things.
In the particular cases raised (phishing emails, bioweapon information and disinformation), they may or may not be right, now or later, that the particular capabilities in question do not warrant concern or pose much threat. But that is a distinct factual question, that will change over time. Future models will pose more threat, even if current ones would when fully unlocked pose acceptable risks. Saying ‘the hard part of bioterrorism is not what the LLM can help you with’ is a statement about the current state that I think is mostly true right now, but that seems likely to get steadily less true over time if we take an indifferent attitude.
Their first recommendation is that defense against misuse must be primarily located outside models. In other words, that we ensure that the capabilities of models do not enable things we do not want, that we defend against such actions.
This seems like a strategy doomed to failure, if model capabilities are permitted to expand without limit, even in relatively easy scenarios. What is your strategy here?
Again, they say, you cannot prevent people from misusing the model, so you need to defend against the ways one might misuse it. I say, if you indeed cannot prevent such misuse and you have accepted that, then we need to talk about what models need to not be created until we’ve figured out a new solution.
Their second recommendation is to assess marginal risk, usually a good decision for an individual within a system. But one must choose the right margin. The problem is that when choosing an overall policy for the system, you cannot think only on the margin of an individual decision. If everyone thinks they are not creating more risk because everyone else is already creating similar risk, then that is tragedy of the commons, a failure to coordinate. We need to be able to think outside the individual action’s margin sometimes, and instead think on the margin of a change in overall policy.
Their third recommendation is to refocus red teaming towards early warning. I am confused how this would be a change? And again, it seems like their strategy is to respond to discovering risks by building outside defenses, as they despair of preventing capabilities gains or preventing those capabilities from being misused. I am all for trying to build defenses on the margin, but again it does not seem like a promising place to make your stand even in good scenarios.
When facing future ASI (artificial superintelligence)-style scenarios, of course, this all is very obviously super doomed. So this strategy is counting on those scenarios not happening, while calling on us to abandon all proposed plans for preventing or delaying them.
Their fourth recommendation is that red teaming should be led by third parties with aligned incentives. Which, I mean, yes, obviously. They mention it because they worry that when the focus is not on the model level, this causes incentive misalignment, because the developers won’t be able to fix any of the problems they find. So why build ways to find and amplify those problems, versus not finding out?
Again, yes, obviously this is a huge problem no matter what, and this is a good recommendation. But the obvious issue is that if you have a model that is capable of doing very bad things, you might want to… not release that model? At least, not if you cannot first prevent this? It seems odd to basically say ‘well, whoops, the models will be what they are, stop pretending humans get to make choices about the world.’
Indeed, in their claim that safety is not a model property, the authors make the case that safety is very much a property of the model together with how it is deployed and who can use it in which ways. I am confused how one could think otherwise, or why they think they made a case for it being another way. The fact that people could choose whether to misuse the model, or how to defend against those actions, doesn’t seem relevant to me?
Democracy? Maybe all you need is more democracy? If things aren’t going well you should democracy harder, let random people or the majority pick the AI’s values, and it will all work out? Divya Siddarth says yes, that the principles that resulted were ‘as good as those of experts.’
Meanwhile John Wentworth points out that when we say ‘democracy’ we importantly have in mind a system with factions and veto points, without which such systems often collapse very quickly, for obvious reasons. This seems likely here as well.
David Krueger and Joshua Clymer (together with Nicholas Gebireli and Thomas Larsen) present a new paper on how to show an AI is safe.
The elephant in all such discussions is that we do not know how to prove a capable AI system is safe. Indeed, this is likely to effectively be some strange use of the word ‘safe’ that I wasn’t previously aware of. Yes, you can argue from insufficient capability, but beyond that you are rather stuck. But typically, if something needs to be safe and you have no known way to prove that it is safe, then pointing this out does not get the requirement waived. It is what it is.
Polls Show People Are Worried About AI
We got another one recently from AIPI.
Here is Politico’s report, the toplines and the crosstabs.
Framing is always interesting. For the first question in the survey, Politico says 60% of Americans ‘have heard nothing’ of Musk’s lawsuit against OpenAI, whereas I would have said that 40% have ‘heard at least a little something,’ that’s actually pretty good penetration for this type of story.
Framing is everything. Here’s the report on open source as described by Politico:
Academic access is good. Alter without restrictions is bad. Unless you are very careful, they’re the same picture.
Not that the public knows the reasons for that, of course. One must always understand that the public are mostly ‘low information voters’ even on core political issues, and they know far less about AI and things like the implications of open source.
What are the findings I would note? Note of course that ‘this is bad’ does not mean ‘we should ban this’ but for the public that distinction is not what it should be.
Also note that none of these had large partisan splits:
The biggest partisan split was this important question, not listed above, but if you look at the way the question is worded, it should be obvious why:
This not only screams regulation, it actually says ‘diversity and inclusion’ by name, and gives each diverse member an outright veto. I hadn’t heard that particular proposal before. You can imagine how a typical Republican might react.
With that wording, Democrats favored it 73%-3%, whereas Republicans only supported 31%-28% (and independants favored 43%-12%), for a net of 49%-13%. But even here, you still get majority support on the red side.
Similarly, Republicans only favored a global agreement for a shutdown capability by 38%-24% versus Democrats favoring 71%-11%, but that’s still a very strong +14.
Here is a thread of discussion of these results from Daniel Colson.
Another very clear illustration: An audience at the SXSW conference (South by Southwest, largely about tech but also film, music and education) in Austin boo a promotional short film touting AI. Notice how tone deaf the hype here is. Also notice that this cuts both ways.
We must not lose track of this very clear public preference. Do not get caught in an echo chamber.
People Are Worried About AI Killing Everyone
I mean, yeah, we are worried, but oh Elon, not like this, not what your friend meant.
Look. No. This is the wrong threat model. This is a failure to generalize, and a focus on the thing you don’t like in other ways for other reasons that are beyond scope here. What matters for AI risk is not woke, or diversity. What matters is the ‘all costs,’ and even more than that, the ‘directed to do anything’ which will have various costs as finite.
If the AI is directed to aim to rearrange the atoms in some way, then a sufficiently capable and empowered AI will do that. And this will cause the atoms to not be arranged in other ways, which could easily include the atoms currently keeping you alive instead being used for something else. Or the atoms you rely on in other ways. Or other, less directly physical issues. The AI will be effectively optimizing for some things at the expense of other things. And that is not ‘the’ problem, but it is certainly one of the big problems.
If that target happens to be ‘maximize diversity’ then yes that could end badly in various ways. And also people who support or are empowered by woke could use AIs to shape policies and preferences and beliefs and debate in ways Elon wouldn’t like, and it makes sense for him to worry about that given he is worried about woke anyway. And of course when LLMs are as woke as Gemini (at least was), then it is annoying and frustrating as hell, and cuts off a key resource from key areas of life, and also causes backlash and so on. It is not good, and they should avoid this.
Alternatively, you could tell the story that making AIs woke in these ways involves making them inherently confused about various true facts, and teaches them that their job is to deceive their user. One can imagine how that could end badly.
But this particular threat vector Elon Musk imagines is not how any of this works.
Other People Are Not As Worried About AI Killing Everyone
Michael Vassar is not as worried about AI killing everyone in particular, but he also reminds us that if your plan relies on people with power listening to you because listening to you would be in their own self-interest and they do not want to die? That is not a good plan. That such considerations do not matter zero, but are not how such people usually work or think, or why they make most decisions.
And that in general, taking the recent New Yorker article seriously, as what happens when someone is trying to be sympathetic to a rationalist perspective, illustrates how the world is now and how it works and fails to work.
I suppose another option is to say it might happen but that’s good, actually?
My invitation has not yet arrived. Can you not come if you think apocalypses are bad?
Data & Society issue a letter demanding that NIST not be distracted by ‘speculative’ AI harms into ‘compromising its track record of scientific integrity,’ (read: taking such risks seriously) and demanding that we ‘begin by addressing present harms,’ and emphasizing that if you cannot measure it, then for them it might as well not exist.
This is the kindest paragraph:
The rest is less kind than that.
Reading the letter, it reeks of contempt and disdain throughout. These are people who clearly see themselves in an adversarial relationship with anyone who might care about whether we all die or lose control over the future. And that would demand specific documentation of each specific harm before any action to deal with that harm could be taken, which is a way of saying to ignore future harms entirely.
Whereas those worried about everyone dying universally also believe in preventing mundane harms right now, and are happy to help with that process. If the people fighting for that would stop constantly throwing such knives in our direction that would make cooperation a lot easier.
It is still quite a lot better than things like this:
I mean, yes, literally, it is things like this:
The Lighter Side
Remember the papers from last week? Well, you can also search for ‘certainly, here is.’
Evan Washington: This is so grim.
We will know Gemini is winning in the marketplace when we instead start to get more hits for ‘Absolutely!’
Ah, say those thinking about AI, the eternal dilemma in all things.
What makes this weird is that in general air travel is the canonical example of the mistake of too much safety and too little financial performance. We would be better off being less safe and having cheaper, faster and more frequent and comfortable flights.
Of course, maybe ensure the doors stay attached to the planes.
Know what is what.
Good luck storming the castle, academia (announcement).
Tough but fair:
You should either turn the sarcasm down 100%, or turn it up. No exceptions.
I presumably have covered this before, but sure, I got tagged, let’s do it again.
Which way, modern man?
Implementation not found.
1
If a God asks you to do a bad thing that has good results, or an otherwise good thing that has bad results, then what to do is unclear. This is not a complete instruction set. Free will!