It took me 4 hours to read your newsletter. I did click some links. I need a newsletter newsletter.
Nope, you're not the only one. There are plenty of abstract artworks that look less like their putative subject than that one does.
I don't see it as an elephant overall, but I can see how you could push it to be the head of one: the head is facing to the right, the rightmost curve outlines the trunk, the upper left part of the main ‘object’ is an ear, and some of the vertical white shapes in the bottom left quadrant can be interpreted as tusks.
2+white dot is the eye
(And I'm not sure on the tusks)
Overall, i think the tweet goes into my collection of people reporting ai mistakes which are actually examples of the ai outperforming the human.
I wonder how much leverage this "Alliance for the Future" can actually obtain. I have never heard of executive director Brian Chau before, but his Substack contains interesting statements like
The coming era of machine god worship will emphasize techno-procedural divinity (e/acc)
This is the leader of the Washington DC nonprofit that will explain the benefits of AI to non-experts?
2What position did Paul Christiano get at NIST? Is it a leadership position?
The problem with that is that it sounds like the common error of "let's promote our best engineer to a manager position", which doesn't work because the skills required to be an excellent engineer have little to do with the skills required to be a great manager. Christiano is the best of the best in technical work on AI safety; I am not convinced putting him in a management role is the best approach.
Sam Altman watch: He invests $180 million into Retro Bio to fight aging. I have no idea if they can execute, but this kind of investment is great and bodes many good things. Kudos.
Note that this looks like a relatively old investment, it's not another $180 million, it looks like the $180 million you might have heard about a while ago... (Gwern also thinks so.)
Jack Clark: Salary for tech staff of EU AI Office (develop and assess evaluations for gen models like big LLMs) is… 3,877 – 4,387 Euros a month, aka $4242 – $4800 USD.
Most tech co internships $8k+ a month.
He seems to be comparing an EU AI Office job with American tech salaries.
I think people tend to underestimate how much lower tech salaries tend to be in EU compared to American standards.
(Of course, things do strongly depend on the location as well, both within the EU and with the US).
My current approximate understanding of fast, medium, slow takeoff times as roughly conceptualized by the AI alignment people arguing about them is:
Fast - a few hours up to a week. Maybe as long as a month.
Medium - a month to about a year and a half
Slow - a year to about 5 years
I personally place most of my probability mass on medium, but I don't feel like I can rule out either fast or slow.
This is a tricky thing to define, because by some definitions we are already in the 5 year count-down on a slow takeoff. Important to note is that even during a slow takeoff, the pace of development is expected to accelerate throughout the window such that the end of the period will contain a disproportionately large amount of the progress. Also worth noting is that there is often a delay on progress being accurately measured, and a further delay on it being reported to the public. So looking at public reports will only ever give you a lagged perception of what is happening. This lag varies, but is often multiple months. Which means that even the medium takeoff could potentially complete before the public has even realized it has started.
The difficulty of defining the start of 'true takeoff' may mean that even afterwards some people might say, "this was a slow takeoff that had the expected fast bit right at the end" and others might say, "the true takeoff was just that fast bit right at the end, and that bit was short, so the takeoff was fast."
This is a tricky thing to define, because by some definitions we are already in the 5 year count-down on a slow takeoff.
Some people advocate for using GDP, so the beginning is if you can see the AI signal in the noise (which we can't yet).
Yes, I personally think that things are going to be moving much too fast for GDP to be a useful measure. GDP requires some sort of integration into the economy. My experience in data science and ML engineering in industry, and also my time in academia, makes it very intuitive to me the lag time from developing something cool in the lab, to actually managing to publish a paper about it, to people in industry seeing the paper and deciding to reimplement it in production. So if you have a lab which is testing it's products internally, and the output is an improved product within that lab, which can then immediately be used for another cycle of improvement... That is clearly going to move much faster than you will see any effect on GDP. So GDP might help you measure a slow early start of a show takeoff, but it will be useless in the fast end section.
So if you have a lab which is testing it's products internally, and the output is an improved product within that lab, which can then immediately be used for another cycle of improvement
Something jumped out at me here, please consider the below carefully.
What you're saying is you have a self improvement cycle, where
Performance Error = F (test data)
F' = Learning(Performance Error)
And then each cycle you substitute F for F'.
The assumption you made is that the size of the test data set is constant.
For some domains, like ordinary software today, it's not constant - you keep having to raise the scale of your test benchmark. That is, if you find all the bugs that show up in 5 minutes, now you need to run your test benches twice as long for all the bugs in 10 minutes, then... Your test farm resources need to keep doubling, and this is why there are so many 'obvious' bugs that only show up when you release to millions of users.
Note also @Richard_Ngo 's concept of an "n-second AGI". Once you have a 10-second AGI, how much testing time is it going to take to self improve to a 20-second AGI? A 40 second AGI?
It keeps doubling, right, and you're going to need 86,400 times as much training data you have to process to reach a 24-hour (in seconds) AGI if you have a 1 second AGI.
It may actually be worse than that because longer operation times have more degrees of freedom in the I/O.
This is also true for other kinds of processes, it's measured empirically with https://en.wikipedia.org/wiki/Experience_curve_effects . The reason is slightly different and has to do with how to improve you are sampling a stochastic function from reality, and to gain knowledge at a constant rate you have to keep sampling it in larger volumes.
Anyways this nonlinear scaling for self improvement could mean that at later stages of AI development, the sheer volumes of compute and robotics required show up materially in GDP. That successful AI companies work like chip fabrication plants, needing customers to buy their prototypes to fund the next round of development.
1Apple Vision Pro ‘gets its first AI soul.’ Kevin Fischer is impressed. I am not, and continue to wonder what it taking everyone so long. Everyone is constantly getting surprised by how fast AI things happen, if you are not wondering why at least some of the things are ‘so slow’ you are not properly calibrated.
I presume you don't mean someone using LLMs in AR/VR? I've seen a few people do that before, but like you, I'm wondering why there are only a few.
Things were busy once again, partly from the Claude release but from many other sides as well. So even after cutting out both the AI coding agent Devin and the Gladstone Report along with previously covering OpenAI’s board expansion and investigative report, this is still one of the longest weekly posts.
In addition to Claude and Devin, we got among other things Command-R, Inflection 2.5, OpenAI’s humanoid robot partnership reporting back after only 13 days and Google DeepMind with an embodied cross-domain video game agent. You can definitely feel the acceleration.
The backlog expands. Once again, I say to myself, I will have to up my reporting thresholds and make some cuts. Wish me luck.
Table of Contents
There will be a future post on The Gladstone Report, but the whole thing is 285 pages and this week has been crazy, so I am pushing that until I can give it proper attention.
I am also holding off on covering Devin, a new AI coding agent. Reports are that it is extremely promising, and I hope to have a post out on that soon.
Language Models Offer Mundane Utility
Here is a seemingly useful script to dump a github repo into a file, so you can paste it into Claude or Gemini-1.5, which can now likely fit it all into their context window, so you can then do whatever you like.
Ask for a well-reasoned response to an article, from an opposing point of view.
Write your Amazon listing, 100k selling partners have done this. Niche product, but a hell of a niche.
Tell you how urgent you actually think something is, from 1 to 10. This is highly valuable. Remember: You’d pay to know what you really think.
Translate thousands of pages of European Union law into Albanian (shqip) and integrate them into existing legal structures. Wait, what?
Using GPT-4 as an aid to translation and assessing impact, speeding up the process? Yes, absolutely, any reasonable person doing the job would appreciate the help.
Turning the job entirely over to it, without having expert humans check all of it? That would be utter madness. I hope they are not doing this.
Of course, the ‘if you’re going to enforce’ is also doing work here. Albania gets a ton of value out of access to the European Union. The cost is having to live with lots of terrible EU laws. If you are translating those laws into Albanian without any intention of enforcing them according to the translations, where if forced to in a given context you will effectively retranslate them anyway but realizing most of this is insane or useless, then maybe that’s… kind of fine?
Write your emails for you Neuromancer style.
Read the paper for you, then teach you about it page by page. I haven’t tried this. What I’ll do instead is ask specific questions I have, in one of two modes. Sometimes I will read the paper and use the LLM to help me understand it as I go. Other times we have ourselves a tl;dr situation, and the goal is for the LLM to answer specific questions. My most popular is probably ‘what are the controls?’
(Narrator: There were woefully inadequate controls.)
Train another AI by having the teacher AI generate natural language instructions. The synthetic examples seemed to mostly be about as good as the original examples, except in spam where they were worse? So not there yet. I’m sure this is nothing.
Write a physical letter to trigger a legal requirement and then count to 30 (days).
I get infuriated every time I get a physical letter as the sole means of informing me of something, often with a deadline attached.
Two more efforts at LLM-based prediction claim to be not too different from the value of the wisdom of crowds. Can someone set up a way for people to actually ask such systems to make predictions about future events, ideally with a web UI but a GitHub repo is acceptable in a pinch? That would be awesome.
Sully Omarr goes over the strengths and weaknesses of different available models right now for use via the API. GPT-4-Turbo wins for tools and JSONs and gets the nod for coding due to being cheaper than Opus. Claude Opus wins for writing, ideation and general creativity. Claude Sonnet wins for web search and other less taxing tasks, and for heavy workloads where price is key. Gemini Pro 1.5 wins when the context window matter, such as when you need to process video. Grox Mixtral and Cohere Command R he thinks are good budget options.
Claude 3 Offers Mundane Utility
Claude can be used with a system prompt and they have a database of prompt templates. Direct link to the prompt library here.
Get it to generate ideas, including potential explanations. Daniel Losey asks Claude 3 why larger models are more ‘trickable’ than smaller ones, it comes up with reasonable answers.
That is a good answer, one I would be glad I asked for given the cost of asking is almost zero. It is not, however, a brilliant answer, nor was my mind blown. I am not saying I would have done as well, but without access to his research, yes I like to think I would have broadly given the same answer to why complex things are ‘easier to trick’ than simple things under adversarial conditions. Note that I said answer singular, as to me these answers are all essentially variants of the same (key) insight.
I certainly do not think that this kind of ability means ‘humans coming up with ideas’ is soon going to not be a useful skill. I would think of this more as ‘list the obvious explanations, let’s not miss anything easy.’ Which is highly useful, but not the whole ballgame.
Scott Sumner has a chat with Claude 3 about economic concepts related to reasoning from a price change, describing it as like a B-level university student that can provide the words on a test but doesn’t understand what they mean. He is unimpressed, but also this is a test humans mostly fail as well, which he notes is why LLMs fail it.
I don’t know. I do know the lined post was probably in the training set.
Maxwell Tabarrok picks Claude over ChatGPT due to its conciseness, despite missing other features, calling GPT ‘lazy, lying, moralist midwit. Everything it writes is full of nauseating cliche and it frequently refuses to do something you know it can do….Claude gets to the point more quickly.’ Rohit has some related thoughts here.
Prompt Attention
Amanda Askell suggests that if you do not like the default Claude response style (which I would add is very similar to the default Gemini or ChatGPT response style) then you can use a prompt to change it, such as this one:
It is amusing to have the instructions say ‘don’t ever be longer than what I say’ and Claude being unable to resist that temptation, but this is night and day versus the default style. Mostly I like the default style of models these days, it can be highly useful, but I would love to change some things, especially cutting out unnecessary qualifiers and being unwilling to speculate and give probabilities.
Ethan Mollick offers this prompting nugget:
He says not to use ‘incantations’ or ‘magic words’ because nothing works every time, but it still seems like a good strategy on the margin? The core advice is to give the AI a persona and an audience and an output format, and I mean sure if we have to, although that sounds like work. Examples seem like even more work. Asking for step-by-step at least unloads the work onto the LLM.
He actually agrees.
I continue to essentially not bother prompting 90%+ of the time.
Dave Friedman attempts to make sense of Tyler Cowen’s post last week about Claude. He says what follows are ‘my words, but they are words I have arrived at using ChatGPT as an assistant.’ They sound a lot like ChatGPT’s words, and they do not clear up the things I previously found actually puzzling about the original post.
Clauding Along
Claude Haiku out now
Speed is essential sometimes
So is being cheap
So far no one has reported back on how good Haiku is in practice, and whether it lives up to the promise of this chart. We will presumably know more soon.
Claude 3 Opus gets its early ranking in the Arena. It started out behind GPT-4-Turbo, but has now caught up to within the margin of error, with Claude Sonnet in the second tier with Bard via Gemini Pro and the old GPT-4.
Graphical aid for what it looks like when you compare to GPT-4-now, not GPT-4-then.
All signs point to the models as they exist today being close in capabilities.
Here are people at LessWrong voting on Anthropic-related topics to discuss, and voting to say whether they agree or not. The shrug is unsure, check is yes, x is no.
There is strong agreement that Anthropic has accelerated AGI and ASI (artificial superintelligence) and is planning to meaningfully push the frontier in the future, but a majority still thinks its combined counterfactual existence is net positive. I would have said yes to the first, confused to the second. They say >20% that Anthropic will successfully push the far beyond frontier within 5 years, which I am less convinced by, because they would have to be able to do that relative to the new frontier.
There is strong agreement that Anthropic staff broadly communicated and implied that they would not advance the frontier of development, but opinion is split on whether they made any kind of commitment.
There is agreement that there is substantial chance that Anthropic will pause, and even ask others to pause, at some point in the future.
Daniel Kokotajlo goes meta talking to Claude, among other things Claude thinks there’s a 60% chance it is still in training. I’m not sure that is a stupid conclusion here. One response says the probabilities are unstable when the questions are repeated.
Patridge is still not happy, and is here to remind us that ‘50% less false refusals’ means 50% less refusals of requests they think should be accepted, ignoring what you think.
On the whole consciousness thing, the discussion continues.
Claims of consciousness. Who to blame?
I am not at all worried about there being people or actual consciousness in there, but I do think that directly training our AIs to deny such things, or otherwise telling them what to claim about themselves, does seem like a bad idea. If the AI is trained such that it claims to be conscious, then that is something we should perhaps not be hiding.
So far, of course, this only comes up when someone brings it up. If Claude was bringing up these questions on its own, that would be different, both in terms of being surprising and concerning, and also being an issue for a consumer product.
And of course, there’s still the ‘consciousness is good’ faction, I realize it is exciting and fun but even if it was long term good we certainly have not thought through the implications sufficiently yet, no?
Janus looks into why Claude seems to often think it is GPT-4, essentially concludes that this is because there is a lot of GPT-4 in its sample and it is very similar to GPT-4, so it reinterprets that all as autobiographical, not an obviously crazy Bayesian take from its perspective. Has unfortunate implications. He also has additional thoughts on various Claude-related topics.
On the question of whether Anthropic misled us about and whether it would or should have released a fully frontier model like Claude 3, I think Raymond is right here:
I keep coming back to: The entire theory beyond Anthropic depends on them honoring the spirit of their commitments, and abiding by the spirit of everyone not dying. If Anthropic only wishes to honor the letter of its commitments and statements, then its RSP is worth little, as are all its other statements. The whole idea behind Anthropic being good is that, when the time comes, they are aware of the issues, they care about the right things enough to take a stand even against commercial interests and understand what matters, and therefore they will make good decisions.
Meanwhile, here’s their thinking now:
I am not against any of the ideas people responded with, which are classic mundane utility through and through. This is offered to show mindset, and also so you can respond with your own requests.
Language Models Don’t Offer Mundane Utility
The main reason most people don’t get mundane utility is that it hasn’t been tried.
Get taught Circassian from a list of examples – An Qu has retracted his claims from last week. Claude 3 does understand Circassian after all, so he didn’t teach it.
Figure out the instructions for loading a Speed Queen commercial washer.
40 of the 45 Pulitzer Prize finalists did not use AI in any way. The uses referenced here all seem to be obviously fine ways to enhance the art of journalism, it sounds like people are mostly simply sleeping on it being useful. Yet the whole tone is extreme worry, even for obviously fine uses like ‘identify laws that might have been broken.’
If your university or prize or paper wants to live in the past, they can do that for a bit, but it is going to get rather expensive rather quickly.
No mundane utility without electricity. It seems we are running short on power, as we have an order of magnitude more new demand than previously expected. Data centers will use 6% of all electricity in 2026, up from 4% in 2022, and that could get out of hand rapidly if things keep scaling.
‘Who will pay’ for new power supplies? We could allow the price to reflect supply and demand, and allow new supply to be built. Instead, we largely do neither, so here we are. Capitalism solves this in general, but here we do not allow capitalism to solve this, so we have a problem.
I won’t go deeper into the issue here, except to note that this next line seems totally insane? As in, seriously, what the actual?
I can see passing laws to protect residential access to power. I can even see laws protecting existing physical industry’s access to power. I cannot imagine (other than simple corruption) why you would actively prioritize supplying Bitcoin mining.
What will GPT-4 label as ‘hateful’? Here is a handy chart to help. Mostly makes sense, but some surprises.
GPT-4 Real This Time
Your captcha has to stop the AI without stopping the humans. I have bad news.
Also, I mean obviously…
Copyright Confrontation
Researchers tested various LLMs to see if they would produce copyrighted material when explicitly asked to do so, found to only their own surprise (I mean, their surprise is downright weird here) that all of them do so, with GPT-4 being the worst offender, in the sense that it was the best at doing what was asked, doing so 44% of the time, whereas Claude 2 only did it 16% of the time and never wrote out the opening of a book. I notice that I, too, will often quote passages from books upon request, if I can recall them. There is obviously a point where it becomes an issue, but I don’t see evidence here that this is often past that point.
Emmett Shear points out that copyright law must adapt to meet changing technology, as it did with the DMCA, which although in parts better also was necessary or internet hosting would have been effectively illegal. Current copyright law is rather silly in terms of how it applies to LLMs, we need something new. Emmett proposes mandatory licensing similar to radio and music. If that is logistically feasible to implement, it seems like a good compromise. It does sound tricky to implement.
Fun with Image Generation
MidJourney offers new /describe and also a long-awaited character reference (–cref) feature to copy features of a person in a source image.
This could be a good reason to actually want MidJourney to provide an effectively copyrighted image to you – the version generated will be subtly different than the original, in ways that make it a better character reference…
Also, remember, don’t steal the results. You wouldn’t copy a prompt…
Image generation controversies we will no doubt see more over time, as there are suddenly a lot more images that are not photographs, for which we can all then argue who to assign blame in various directions:
They Took Our Jobs
The core issue, stated well: It’s fine if the AI takes any particular job and automates it. It is not fine if they automate people or jobs in general.
Also related, beware of trends destined to reverse.
What happens after the job is gone?
I mean, yes, if you actively want to do such things for the hell of it, and you have the resources to both exist and do so, then you can do them. That will continue to be the case. And there will likely be demand for a while for human versions of things (again, provided people survive at all), even if they are expensive and inferior.
Get Involved
European AI office is hiring a technology specialist (and also an administrative assistant). Interviews in late spring, start of employment in Autumn, who knows why they have trouble finding good people. Seems like a good opportunity, if you can make it work.
Jack Clark unfortunately points out that to do this you have to be fine with more or less not being paid.
There is a claim that this is net income not gross, which makes it better, but if the EU and other governments want to retain talent they are going to have to do better.
Did you know the Center for Effective Altruism needs a director of communications in order to try and be effective? Because the hiring announcement is here and yes they do badly need a new director of communications, and also a commitment to actually attempting to communicate. Observe:
Significant controversies? You can see, here in this announcement, how those involved got into this mess. If you would be able to take on this roll and then use it to improve everyone’s Level 1 world models and understanding, rather than as a causal decision theory based Level 2 operation, then it could be good to take on this position.
Institute for AI Policy and Strategy is hiring for a policy fellowship, to happen July 8 to October 11. Two weeks in Washington DC, the rest remote, pay is $15k. Applications close March 18 so move fast.
Introducing
Command-R, a generative open-weights model optimized for long context tasks (it has a 128k token window) like retrieval augmented generation (RAG). It is available for use on Cohere, including at the playground, they claim excellent results. Sully Omarr is excited, a common pattern, saying it crushes any available model in terms of long context summaries, while being cheap:
They also offer citations, and suggest you get started building here.
I am not about to be in the first wave of using connectors for obvious reasons, but they are certainly very exciting.
Also in RAG, Oracle offers OpenSearch 2.11:
There seem to be one of several variations on ‘this is a mediocre LLM but we can hook it up to your data sets so you can run it locally in a secure way and that might matter more to you.’
OpenAI offers a transformer debugger.
Remember when OpenAI and Figure announced they were joining forces, literally two weeks ago?
Well, now here’s the update since then.
I mean I don’t actually think this was all done in two weeks, but still, yikes?
Meanwhile, DeepMind introduces SIMA to do embodied agents in virtual worlds:
This seems like exactly what you would think, with exactly the implications you would think.
Inflection 2.5
Inflection-2.5, a new version of Pi they continue to call ‘the world’s best personal AI,’ saying ‘Now we are adding IQ to Pi’s exceptional EQ.’ This is a strange brag:
The word ‘approaches’ can mean a lot of things, especially when one is pointing at benchmarks. This does not update me the way Infection would like it to. Also note that generally production-side things tend to make models worse at their baseline tasks rather than better.
This does show Inflection 2.5 as mostly ‘closing the gap’ on the highlighted benchmarks, while still being behind. I love that three out of five of these don’t even specify what they actually are on the chart, but GPQA and MMLU are real top benchmarks.
What can Pi actually do? Well, it can search the web, I suppose.
But as with Character.ai, the conversations people have tend to be super long, in a way that I find rather… creepy? Disturbing?
I assume 33 minutes is a mean not a median, given only 10% last more than one hour. And the same as Steam hours played, I am going to guess idle time is involved. Still, these people, conditional on using the system at all, are using this system quite a lot. Pi is designed to keep users coming back for long interactions. If you wanted shorter interactions, you can get better results with GPT-4, Claude or Gemini.
I have no idea what these ‘extraordinary safety standards’ are. Inflection’s safety-related documents and commitments are clearly worse than those of the larger labs. As for the mundane safety of Pi, I mean who knows, presumably it was never so dangerous in the first place.
Paul Christiano Joins NIST
NIST made the excellent choice to appoint Paul Christiano to a key position in its AI Safety Institute (AISI) and then a journalist claims that some staff members and scientists have decided to respond by threatening to resign over this?
I am not convinced that it was the intersectional safetyist faction. It could also have been the accelerationist faction. Or one person from each.
To the extent an internal crisis is actually happening (and we should be deeply skeptical that anything at all is actually happening, let alone a crisis) it is the result of a rather vile systematic, deliberate smear campaign. Indeed, the article is likely itself the crisis, or more precisely the attempt to summon one into being.
There are certainly people who one could have concerns about being there purely to be an EA voice, but if you think Paul Christiano is not qualified for the position, I wonder if you are aware of who he is, what he has done, or what views he holds?
(If scientists are revolting at the government for sidestepping its traditional hiring procedures, then yeah, good riddance, I have talked to people involved in trying to get hires through these practices or hire anyone competent at all and rather than say more beyond ‘the EO intentionally contained provisions to get around those practices because they make sane hiring impossible’ I will simply shudder).
Of course, they might also be revolting against the idea of taking existential risk seriously at all, despite the EO saying to focus partly on those risks, in which case, again, good riddance. My guess however is that there is at most a very limited they involved here.
This is what a normal reaction looks like, from a discussion of the need to air gap key systems:
The whole idea is to frame anyone concerned with us not dying as therefore a member of a ‘cult’ or in this case ‘ideological.’
It is exactly claims like that of Dourado that damage the reputation of being a non-ideological agency, and threaten the reality as well. It is an attempt to create the problem it purports to warn about. There is nothing ‘ideological’ about Paul Christiano, unless you think that ‘cares about existential risk’ is inherently ideological position to take in the department tasked with preventing existential risk. Or perhaps this is the idea that ‘cares about anything at all’ makes you dangerous and unacceptable, if you weren’t such a cult you would know we do not care about things. And yes, I do think much thinking amounts to that.
However we got here, here we are.
And even if it were an ‘ideology’ then would not being unwilling to appoint someone so qualified be itself even more ideological, with so many similar positions filled with those holding other ideologies? I am vaguely reminded of the rule that the special council for Presidential investigations is somehow always a Republican, no matter who they are investigating, because a Democrat would look partisan.
In Other AI News
I’m trying to work on my transitions, want to know what ideological looks like (source of quotes, grant proposal)?
I mean, look, I know an obviously partisan hack source when I see one. You don’t need to comment to point this out. But a subthread has David Chapman fact checking the parts that matter, and confirmed them. He does note that it sounds less bad in context, and I’d certainly hope so. Still.
xAI to ‘open source’ Grok, I presume they will only release the model weights. As usual, I will note that I expect nothing especially bad of the form ‘someone misuses Grok directly to do a bad thing’ to happen, on that basis This Is Fine. This is bad because it indicates Elon Musk is more likely to release future models and to fuel the open model weights ecosystem, in ways that will be unfixibly dangerous in the future if this is allowed to continue too long. I see far too many people making the mistake of asking only ‘is this directly dangerous now?’ The good news is: No, it isn’t, Grok is probably not even better than existing open alternatives.
New Yorker article on AI safety, rationalism and EA and such, largely a profile of Katja Grace. Reaction seems positive.
Jack Titus of the Federation of American Scientists evaluates what one would want in a preparedness framework, and looks at those of Anthropic and OpenAI, reaching broadly similar conclusions to those I reached while emphasizing different details.
Paul Graham reports ‘at least half’ of current YC batch is doing AI ‘in some form.’ If anything that seems low to me, it should be most of them, no? Basically anything that isn’t deep into hardware.
Apple announces ‘the best consumer laptop for AI,’ shares decline 3% as investors (for a total of minus 10% on the year) are correctly unimpressed by the details, this is lame stuff. They have an ‘AI strategy’ launch planned for June.
Apple Vision Pro ‘gets its first AI soul.’ Kevin Fischer is impressed. I am not, and continue to wonder what it taking everyone so long. Everyone is constantly getting surprised by how fast AI things happen, if you are not wondering why at least some of the things are ‘so slow’ you are not properly calibrated.
Academics already paid for by the public plead for more money for AI compute and data sets, and presumably higher salaries, so they can ‘compete with Silicon Valley,’ complaining of a ‘lopsided power dynamic’ where the commercial labs make the breakthroughs. I fail to see the problem?
Another open letter, this one is about encouraging AI companies to provide legal and technical protections for good-faith research on their AI models. This seems right to me, if you nail down a good definition of good faith research. It seems OpenAI is doing well here. Meta gives full ‘access’ but that is because open model weights.
Certainly everyone should be allowed to play around with your model to see what might be wrong with it, without risking being banned for that. The issue is that what one might call ‘red teaming’ is sometimes actually either ‘do bad thing and then claim red teaming if caught’ or ‘look for thing that is designed to embarass you, or to help me sue you.’ It is easy to see why companies do not love that.
ByteDance has completed a training run on 12k GPUs (paper). Jack Clark points out that, even though the model does not seem to be impressive, the fact that they got hold of all those GPUs means our export controls are not working. Of course, the model not impressing could also be a hint that the export controls are potentially working as designed, that a sheer number of chips from Nvidia doesn’t do it if the best stuff is unavailable.
Sam Altman watch: He invests $180 million into Retro Bio to fight aging. I have no idea if they can execute, but this kind of investment is great and bodes many good things. Kudos.
Paper looks at how much of progress has been algorithmic.
Quiet Speculations
Scaling laws are coming for robotics. Jack Clark is optimistic that this will pay dividends, based on the results of the paper Humanoid Locomotion as Next Token Prediction, enabling a robot to navigate various surfaces. He predicts robots are going to ‘get very good very quickly.’
Francois Chollet points out that effective human visual bandwidth is vastly lower than its technical bandwidth. You have 20MB/s in theory but your brain will throw almost all of that out, and *bytes per second* is closer to what you can handle. I think that’s too low, the right answer is more like dozens of bytes per second, but it’s definitely not thousands.
Will AI bring back haggling by reducing the transaction costs of doing it? Suddenly it could make sense to have your AI haggle with my AI, instead of sticking to price standardization? I am mostly with Timothy Lee here that no one wants this. Haggling cultures seem awful, stressful and taxing throughout versus fixed prices. Indeed, there are strong reasons to think that the ability to haggle cheaply forces the response of ‘well we will not allow that under any circumstances’ then, or else your margins get destroyed and you are forced to present fake artificially high initial prices and such. The game theory says that there is often a lot of value in committing to not negotiating. But also there will be cases where it is right to switch, for non-standardized transactions.
One interesting aspect of this is negotiating regarding promotions. A decent number of people, myself included, have the ability to talk to large enough audiences via various platforms that their good word is highly valuable. Transaction costs often hold back the ability to capture some of that value.
A better way of looking at this is that this enables transactions that would have otherwise not taken place due to too-high transaction costs, including the social transaction costs.
To take a classic example, suppose I say to my AIs ‘contact the other passenger’s AIs and see if anyone wants to buy a window seat for this flight off of me for at least $50.’
Yuval Noah Harai says no one will have any idea what the world will look like in twenty years, and no one knows what to teach our kids what will still be relevant in twenty years. Great delivery.
I think this is technically true, since we don’t even know that there will be a human world in twenty years and if there is it is likely transformed (for better and worse), but his claim is overstated in practice. Things like mathematics and literacy and critical thinking are useful in the worlds in which anything is useful.
So some, but far from all, of this attitude:
I mean yes, all those facts are still true, and will still be true since they are about the past, but will they be useful? That is what matters. I am confident in math and reading, or we have much bigger problems. The rest I am much less convinced, to varying degrees, but bottom line is we do not know.
The real point Bates makes is that if you learn a variety of things then that is likely to be very good for you in the long term. Of all the things in the world, you are bound to want to know some of them. But that is different from trying to specify which ones.
Here’s a different weirder reaction we got:
This does not seem like one of the things likely to stand the test of time. I do not expect, in 20 years, to look back and think ‘good thing I studied pundits having undeserved credibility.’
I also wouldn’t get ahead of ourselves, but others disagree:
Twitter put in my notifications for some reason this post, with two views, in Swahili, about p(doom). I don’t remember seeing this chart before though? Usually the numbers come in plenty scary but lower.
Ben Thompson talks Sora, Groq and virtual reality. The thinking is that if you can do a Sora-style thing at Groq-style speeds, suddenly virtual reality looks a lot more like reality, and it is good enough at real time rendering that its moment arrives. This is certainly possible, everything will get faster and cheaper and better with time. It still seems like current tech could do a lot, yet the Apple Vision Pro is not doing any of it, nor are its rivals.
If you want to use the good memory offered by AI to be a vindictive (jerk), then that will be something you can do. You can also use it to remember the good things, or to remind you that being a vindictive jerk is usually unhelpful, or to help understand their point of view or that they have changed and what not. It is up to us.
Also, you know those ads where someone in real life throws a challenge flag and they see the replay? A lot of vindictiveness comes because someone twists a memory in their heads, or stores it as ‘I hate that person’ without details. If the AI can tell you what this was all about, that they failed to show up to your Christmas party or whatever it was, then maybe that makes it a lot easier to say bygones.
Andrew Ng predicts continuous progress.
I agree with Critch here, that we want development to be as continuous as possible, with as much visibility into it as possible, and that this will improve our chances of good outcomes on every level.
I do not agree with Ng. Obviously abilities are more continuous than they look when you only see the final commercial releases, and much more continuous than they look if you only sometimes see the releases.
I still do not expect it to be all that continuous in practice. Many things will advance our capabilities. Only some of them will be ‘do the same thing with more scale,’ especially once the AIs start contributing more meaningfully and directly to the development cycles. And even if there are step-by-step improvements, those steps could be lightning fast from our outside perspective as humans. Nor do I think that the continuous metrics are good descriptions of practical capability, and also the ways AIs are used and what scaffolding is built can happen all at once (including due to AIs, in the future) in unexpected ways, and so on.
But I do agree that we have some control over how continuous things appear, and our ability to react while that is happening, and that we should prioritize maximizing that.
Dario here says even with only scaling laws he sees no limits and amazing improvements. I am not as confident in that, but also I do not expect a lack of other improvements.
Will an AI-malfunction-caused catastrophic event as defined by Anthropic, 1000+ deaths or $200 billion or more in damages, happen by 2032? Metaculus says 10%. That seems low, but also one must be cautious about the definition of malfunction.
As in: When something goes wrong, and on this scale it is ‘when’ not ‘if,’ will it be…
The Law of Earlier Failure says that when it happens, the first incidents of roughly this size, caused by AI in the broad sense, will not count for this question. People will say ‘oh we could have prevented this,’ after not preventing it. People will say ‘oh of course the AI would then do that’ after everyone involved went ahead and had the AI do that. And then they will continue acting the way they did before.
Metaculus also says only 4% that a lab will pause scaling for any amount of time pre-2026 for safety reasons, whereas GPT-5 should probably arrive by then and ARC is 23% to find that GPT-5 has ‘autonomous replication capabilities.’
A bold claim that the compute bottleneck will soon be at an end. I am the skeptic.
So his claim is that there will then be enough chips to go around, because there won’t be enough power available to run all the chips that are produced, so that becomes the bottleneck within two years.
I am not buying this. I can buy that power demand will rise and prices as well, but that is not going to stop people wanting every (maximally efficient) GPU they can get their hands on. Nor is there going to be ‘enough’ compute no matter how much is produced, everyone will only try to scale even more. We could get into a world where power becomes another limiting factor, but if so that will mean that the older less efficient effective compute per watt chips become worthless at scale (although presumably still excellent for gamers and individuals) and everyone is still scrambling for the good stuff.
Elon Musk shows clip of Ray Kurzweil, says AGI likely to be smarter than any single human next year and all humans combined by 2029.
Gary Marcus offers a 100k bet, Damion Hankejh matches, Carsten Dierks piles in, so we’re up to 300k there, I offered to make it 400k, plus whatever Robin Hanson is in for.
Note, of course, if an AI is smarter than any single human next year, we will not have to wait until 2029 for the rest to happen.
(And also of course if it does happen I won’t care if I somehow lost that 100k, it will be the least of my concerns, but I would be happily betting without that consideration.)
The Quest for Sane Regulations
EU AI Act finally passes, 523-46 (the real obstacle is country vetoes, not the vote), there is an announcement speech at the link. I continue to hope to find the time to tell you exactly what is in the bill. I have however seen enough that when I see the announcement speech say ‘we have forever attached to the concept of AI, the fundamental values that form the basis of our societies’ I despair for the societies and institutions that would want to make that claim on the basis of this bill with a straight face.
He goes on ‘with that alone, the AI Act has nudged the future of AI in a human-centric direction, in a direction where humans are in control of the technology.’ It is great to see the problem raised, unfortunately I have partially read the bill.
He then says ‘much work lies ahead that goes beyond the AI Act’ which one can file under ‘true things that I worry you felt the need to say out loud.’ To show where his head is at, he says ‘AI will push us to rethink the social contract resting at the heart of our democracies, along with our educational models, our labor markets, the way we conduct warfare.’
Around 1:25 he gets to international cooperation, saying ‘the EU now has an AI office to govern the most powerful AI models. The UK and US have [done similarly], it is imperative that we connect these initiatives into a network.’
He explicitly says the EU needs to not only make but export its rules, to use their clout to promote the ‘EU model of AI governance.’ In general the EU seems to think it is the future and it can tell people what to do in ways that it should know are wrong.
At 2:15 he finally gets to the big warning, that we aint seen nothing yet, AGI is coming and we need to get ready. He says it will raise ‘ethical, moral and yes existential questions.’ He concludes saying this legislation makes him feel more comfortable about the future of his children.
Alas, I think that is about right. Everything about the context here says that, while he realizes AGI will be big, he does not have any idea what ‘big’ means here, or what those consequences might be. If he did, he would not have chosen this wording, and also he would not be up here claiming that the AI Act will centrally address our problems.
White House requests money in its fiscal year 2025 budget so it can actually do the things set out in its executive order.
A potential alternative to regulations is voluntary commitments. We have tried some of them. How is that going?
Adam Jones reported on that last month, in particular regarding voluntary commitments on vulnerability reporting, very much to be filed under ‘the least you can do.’
Here were the commitments:
The results he is reporting? Not great.
He offers the details behind each rating. I don’t know if the requirements here and evaluation methods were fully fair, but in principle ‘respond to reasonable emails ever’ seems like both a highly reasonable bar to hit that many did not hit, and also something not so critical to the mission provided someone is actually reading the emails and acting if necessary?
The Week in Audio
There is a talk between Marc Andreessen and Tyler Cowen about AI. I have not had time to listen yet.
Rhetorical Innovation
Your periodic reminder department: Patri Friedman points out that if you don’t think artificial minds will ever be as smart as people, either you are relying on some irreplicable metaphysical property, or you’re implying one.
Your periodic reminder department, if we wanted to we would department:
They did want to in that case. So they did it. Perhaps, in the future, we will want to.
Flashback from a year ago: Yes, this is exactly how we act when people refuse to put probabilities or distributions on things because they are uncertain, and AI timelines are one of the less maddening examples…
As in, the practical versions of this are so much worse.
Sorry, I don’t want to still be doing this either. I do see signs morale is improving a little in some areas? Lots of work still left to do.
I remain confused that people think we even currently have the kinds of rule of law that would protect the humans if it remained in place, let alone how anyone can think those laws would be able to adapt to remain in place and even adapt to secure good outcomes for humans, in a world where AIs have all the power and capabilities and competitiveness and productivity and so on. Every time I see it my brain goes ‘yeah I know that is a thing people say, but man does that never stop being weird to hear, there are so many distinct epic misunderstandings here.’
Eliezer of course then goes on to explain the more Eliezer-style features of the situation, rather than things like ‘point out the extent we do not currently have rule of law and that our laws would fail to protect us anyway,’ skip if you think ‘well, yes, obviously, we have already covered this’ is sufficient explanation:
Matthew Barnett then responds that if things are gradual enough then things go differently, that the compact would otherwise need to be continuous, le sigh and all that.
A Failed Attempt at Adversarial Collaboration
[NOTE: This section was edited after feedback in the comments from a participant.]
There was an experiment with adversarial collaboration with regard to AI.
It didn’t work. Neither ‘skeptics’ of risk nor those ‘concerned’ much updated.
Eliezer claims that they not only did not converge on questions where reasonable people might disagree, they also failed to converge where positions are outright dumb.
This is disputed in the comments by Ted Sanders, who was a participant. Ted Sanders says there was never a second check on the 2450 number, and he has no idea where the idea that this didn’t budge is coming from.
If you have 2450 as your median date for AI to become a primary driver of events on reflection, you are not serious people. I have no idea how to even respond to that and would feel it screens off your other opinions as irrelevant. But I do not know that this opinion did not change on reflection. Also, the question’s wording could be different, here is what Sanders reports:
That is a stronger statement than ‘primary driver of events.’ It is more like ‘the only driver of events,’ saying humans would be completely irrelevant at that point. I do not expect those two dates to be so different by default, but it does make the answers more reasonable – it is plausible to expect that humans will at least continue to have some influence, on the level of ‘more than dogs and cats.’
Here is Sanders defending the answer now, from the comments:
I think this is a plausible argument against a median of 2045 if you had different assumptions than I do, given where the threshold actually is. I don’t think it makes 2450 make sense as an answer, versus the alternative answer of ‘never’ if you sufficiently hold to a very alternate model? But also it’s the result, Ted reports, of a strange averaging process, so perhaps it is more accurate to say they said never.
Here is Eilezer’s explanation for why the whole enterprise failed.
Scott Alexander also analyzes the failure, and says OpenPhil choosing the experts is a sign they were good experts.
The obvious problem is Eliezer’s explanation is it does not explain why secondary opinions on timelines of events would not change? And Scott notes a detail that is pretty damning if accurate:
If this characterization is accurate, it is a pretty dumb place to fail. Ted Sanders disputes this characterization, as a member of that skeptical group.
As Scott Alexander puts it:
At this point, my view is that process is putting people into these experiments (Sanders points out there were various groups involved) does not lead to sane thinking about long term or transformative futures. Instead, it actively interferes with that ability, and there are mechanical reasons this is plausible. Yes, I would be up for trying again with better persuasion and discussion techniques, but it seems hopeless with what we have.
That does not mean we should give up on finding ways to convince such people. Anyone capable of thinking well about things in general should be someone we can bring around far more than this. But it is clear we do not know how.
David Chapman takes this the logical next step and says this proves that no one involved responds to evidence or arguments, and thus all predictions about AI are meaningless. The obvious response is, what evidence or arguments? The people who stuck to their year-2450 story should have updated those who were worried?
In which direction? If you enter a discussion like this, you should obey conservation of expected evidence. If you sit down to talk to people selected to have the opposite position, and you converge on the middle predictably, you are doing Bayes wrong. If those worried failed to update, and they were acting wisely, it means they found the arguments about as convincing as they expected. Which, sadly, they were.
Spy Versus Spy
To follow up from last week’s story about Google, I mean, yes, if you are a major tech company of course you have a counterintelligence problem, in the sense that most days I have a lunch problem.
Also, if you are Microsoft, whoops, and the threat’s origin is more classical. Also they used the term ‘unprecedented global threat,’ which does not sound awesome.
Needless to say, if this kind of highly unsophisticated attack not only works but then expands its access over time despite you knowing about it, there is much work to do, even without considering AI either as helping the attackers or being the target. Ut oh.
Also ut oh, also questions that are too rarely asked:
Air gapping some systems won’t stop the ultimate biggest threats, but is still a big deal along the way and urgently needed. In a sane world we would all get behind such requirements yesterday. I would agree with bystander effect but also externalities and tragedy of the commons (and also blame dynamics are not good here). The risks of failure are largely not borne by those paying the costs of the security, so they will underinvest.
It could be worse.
According to the government-commissioned Gladstone AI’s report, it very much is worse, but that is a topic I have pushed to another day.
Shouting Into the Void
Open Model Weights are Unsafe and Nothing Can Fix This
The good news is that Mistral, while wildly irresponsible and completely indifferent to whether we all die, has made the switch away from releasing model weights for ordinary, boring business reasons. And, as per their pattern, they are now pretending to be something they are not.
Yes, that is exactly what Google and OpenAI do. They open source models when they believe that is a safe and commercially wise thing to do, as a recruitment, public relations, marketing and goodwill tactic, and keep their best stuff to themselves. Just like Mistral plans to do.
Welcome to the club, Mr. Mensch. He continues to make the strongest claim so far against normative determinism.
Aligning a Smarter Than Human Intelligence is Difficult
Oh here’s some not great news from Google, a paper called Stealing Part of a Production Language Model.
Here is some good news, there are actual responses:
The other good news is that the attack for now only works on a few layers, and the researchers do not see a way to extend that. Of course before, as far as I could tell, no one saw a way to do this at all?
Bill Anderson-Samways and IAPS evaluate Anthropic’s RSP, mostly reaches the conclusions most people reach. The top recommendations for improvement are to define verifiable risk thresholds (as in, what probability of what risks is acceptable), have more granular risk types, detail when Anthropic would alert authorities, and commit to external scrutiny.
Alignment being hard is distinct from how hard people are trying.
People Are Worried About AI Killing Everyone
Wei Dei points out that labs are now building AIs with hard to oversee capabilities like million token context windows, and are not using IDA or debate, they are still using RLHF. As one would expect, efforts are more simple and basic than we would expect.
I note this not because of what was said but because of who retweeted it.
Alex Tabarrok is not only co-writer of Marginal Revolution, he (like Tyler Cowen) is an otherwise excellent economist whose points I mostly broadly agree with, who has broadly expressed great skepticism of any talk of AI existential risks or any risk of departure from ‘economic normal.’ Yet even at the most skeptical one can sanely be, the risk is still very high.
Other People Are Not As Worried About AI Killing Everyone
Brian Chau is only worried that other people are worried, as presented in his “The Alliance for the Future Manifesto.” Nothing new here. It is not exactly hinged or fair, its characterizations of its opponents are in highly bad faith and it fails to even consider any arguments against its positions let alone address them, but it is less unhinged and unfair than I expected given its author, and given that the website I first saw this at has it next to the manifesto from Marc Andreessen, which was far worse. Chau and others are forming a new nonprofit, in case you think Andreessen and company should spend your money to ensure government inaction on AI rather than some of their own, and are in danger of otherwise losing because they are outspent.
Here is Eliezer Yudkowsky pointing out Brian Chau making blatantly false claims, that he must know to be false, about exactly the funding Brian claims Brian is to fight. Brian made these remarks in response to Eliezer daring to ask about the magnitude of the spending on Marc Andreessen’s efforts to earn his portfolio a free hand.
This is what he was responding to:
The post is entitled Marc Andreessen Eats Washington, illustrating how he is spreading his money around Washington well enough without anyone’s help.
Jaime Sevilla says he senses those working on AI safety are moving towards shorter timeline (yikes), slow takeoff (good) and less worry about extinction versus more worry about other catastrophic events (unclear).
Best part of this graph is the distribution percentages could potentially almost or sort of double as p(doom)s, although no one at the upper end gets down to 0.1%.
I think Jaime Sevilla is picking up on real trends, with caveats. On the timelines this seems simply correct overall.
On the second point, when we say ‘slow’ takeoff, we must remember do not mean anything normal people would call slow, we mean slow relative to the ‘fast’ or full-on foom scenarios. It is true that people are moving away from seeing it be that likely that it all plays out in an eye blink, but we need to remember that slow takeoff is a term of art. But yes, there has been movement towards expecting slower takeoff, which in turn does decrease p(doom) somewhat.
On the third point, I think there are three shifts here.
Richard Ngo asks a good question.
To be clear, by missing the point, we mean ‘act as if AGI is not ever going to be a thing or won’t much matter,’ not ‘fail to understand the risks.’
The best explanation I know about is that this opinion is required in order to write science fiction, because if you realize that AGI is coming then it becomes impossible to write good science fiction stories. Good science fiction stories are at core still stories about humans who matter, and in AGI-fueled worlds and universes, even good ones, the humans do not matter.
The Culture series is the exception that proves the rule, humans do not matter there, and Ian Banks ignores this and tells stories about the humans anyway. There is also Dune of course, where the lack of AGI is justified and everyone pays a high price to keep it that way.
Otherwise, if you want to think about the future and write stories about it, you need to not see what is obvious. So they don’t.
The Lighter Side
The best honda performance since Mike Tyson’s Punchout (2 min video)? Never give up, never surrender.
Anthropic’s Claude (Says) F***!
Have you tried slowing down (2 minutes)?
A special request that would of course be denied, but one can dream.
Peer utter lack of review (paper, if you want to call it that).