I do not think that is happening now for MIRI, and I hope not for others either. I strongly agree with Rob that there is a strong social disincentive to extreme doom estimates.
I know from experience that a sufficiently strong social disincentive can actually increase people's tendency to groupthink. Some groups define themselves largely by the hate they get from the outgroup !
I don't think that it's the case for MIRI, and I don't expect them to fall for this trap. But I hope they're keeping this kind of dynamic in mind and checking from time to time if their position isn't made more rigid by the kind of undocumented criticism they often get.
I also had a bunch of thoughts of ‘oh, well, that’s easy, obviously you would just [OH MY LORD IS THIS CENSORED]’
I applaud your Virtue of Silence, and I'm also uncomfortable with the simplicity of some of the ones I'm sitting on.
The "million token" recurrent memory transformer was first published July 2022. The new paper is just an investigation whether the method can also be used for BERT-like encoder models.
Given that there was a ton of papers that "solved" the quadratic bottleneck I wouldn't hold my breath.
If months of debate with superforecasters didn’t accomplish much, that’s really disheartening
I participated in Tetlock's tournament. Most people devoted a couple of hours to this particular topic, spread out over months.
A significant fraction of the disagreement was about whether AI would be transformative this century. I made a bit of progress on this, but didn't get enough feedback to do much. AFAICT, many superforecasters know that reports of AI progress were mostly hype in prior decades, and are assuming that is continuing unless they see strong evidence to the contrary. They're typically not willing to spend much more than an hour looking for such evidence.
I'm excited for the new set of filters that do nothing to change a video or picture except add deepfake artifacts to it. In this way, when damning video gets out, people can just quickly run it through one of these things, and then repost it themselves pointing out the artifacts and claiming the whole thing is fake.
If we had the ability to create one machine capable to centrally planning our current world economy, how much processing power/ memory would it need to have? Interested in some Fermi estimates.
To which I would reply, this is AI-complete, at which point the AI would solve the problem by taking control of the future. That’s way easier than actually solving the Socialist Calculation Debate.
As a data point, Byrne Hobart argues in Amazon sees like a state that Amazon is approximately solving the economic calculation problem (ECP) in the Socialist Calculation Debate when it sets prices on the goods on its online marketplace.
US's 2022 GDP of $25 462B is 116x Amazon's 2022 online store revenue of $220B.
Assuming scaling[1], it would take 500 000x the compute Amazon uses (for its own marketplace, excluding AWS) to approximately solve the ECP for the US economy (to the same level of approximation as "Amazon approximately solves the ECP"[2]).
In practice, I expect Amazon's "approximate solution" to be much more like [3], so maybe a factor of merely 800x.
The economic calculation problem (ECP) is a convex optimisation problem (convexity of utility functions) which can be solved by linear programming in .
Which is not a vastly superhuman level of approximate solution? For example, Amazon is no longer growing fast enough to double every 4 years. Its marketplace also doesn't particularly incentivise R&D, while I think a god-like AI would incentivise R&D.
I just made it up.
There was a Senate Committee on Armed Services hearing on AI, in which the CEO of Palantir urged us to spend 5% of our budget on capabilities that would ‘terrify’ our enemies, and that we need to hook our weapons up to our AI systems as soon as possible, or else ‘China will surpass us in a decade.’
I... honestly have no words. I am in general AI optimist. I am not this-kind-of-humanity optimist. With pro-AI people like that we don't need actual ASI or AGI to blow ourselves. This is insanity.
if I had more time, I’d be using such tools to learn to code rather than to learn to keep not coding.
As someone heavily involved in development for 16 years and ML for 8+ - I heavily endorse this. This is where the actual value lies.
It doesn't actually help with a lot of really difficult an complex tasks, but it raises the "ground level" of tasks you need to worry about without manual involvement. More or less the same what High-level programming languages did before that, and low level, and assembler and ...
There were two big developments this week.
One is that Google merged Google Brain and DeepMind into the new Google DeepMind. DeepMind head and founder Demis Hassabis is in charge of the new division. We will see how this plays out in practice, seems very good for Google.
The other is that a paper claims to have figured out how to extend the context window to one… million… tokens, making the cost per token linear rather than quadratic. If so, and this doesn’t have big other downsides, that’s a huge game. A comment suggests this is more on the 'in mice' level, so perhaps we should not get our hopes up.
Lots of other stuff happened as well, those are the two must-know items.
Split off this week were:
Table of Contents
Language Models Offer Mundane Utility
Did you know the Bing context window in creative mode is about 13,500 tokens?
Your message length limit is 2000 characters, but that is only enforced by the HTML so you can “inspect” the chat box, double click on the “maxlength” parameter and change it to whatever you like. And it will stay changed between sessions. The real max is 10,552 tokens. Presto.
Claim there exists a lot of mundane utility offered.
Suspicion of ‘something like 3’ aside, this is the kind of thing people would be highly inclined to tell Sam Altman, so grain of salt. Still, I can totally buy it.
My own mundane utility update: It’s been relatively slow going, but at this point I’d say Bing plus ChatGPT are providing a substantial productivity boost. Useless for the actual writing, haven’t found a way to edit, but my knowledge seeking and learning has sped up considerably.
You can if you squint kind of rediscover Kepler’s Third Law ‘the way Kepler did it,’ if you know you are looking for such a law and broadly what type of form it will take, and you explicitly tell the AI what method to use, and it probably doesn’t depend on the training set containing the answer.
If anything I feel less nervous knowing this is what it took. The AI isn’t ‘doing the real work’ here at all.
Let you start a ‘business’ on Replit without actual coding experience, by using a combination of bounties and ChatGPT calls and their Ghostwriter. This does not seem like a way to build a great code base – if I had more time, I’d be using such tools to learn to code rather than to learn to keep not coding.
You can have ChatGPT generate exercises and in-class educational activities.
It can tell you the location of the nearest McDonalds without knowing where you are.
Also the AI can pin itself to the top of your Snapchat feed with no way to alter that unless you pay for Snapchat+. So they imposed a feature many people don’t want, and the only way to get rid of it is to pay. There are also many reports of it being ‘creepy’ for various easy to understand reasons.
People aren’t taking kindly, review averages are down quite a lot.
You can have a Facetime conversation with an AI rather than typing. That doesn’t mean you would want to. Voice is agonizingly slow when there is no detail information available, no meaningful nuance or tone, and you’re making small talk? And terrible small talk at that. The audio tech is advancing nicely though, and the face moves nicely.
You have more choices in the dating app message composition game. Washington Post points to Razz, also links to YourMove.ai or potentially the ‘chat bot that is you’ that is personal.ai.
Show a little faith and get rewarded?
Language Models Don’t Offer Mundane Utility
Strangely, also D&D encounters?
Fun With Image, Audio and Video Generation
David Holz, founder of Midjourney, has time for your ethical considerations, but has no time to write up his time for your ethical considerations.
Bring friends to Pepperoni Hug Spot, via this 30 second AI-generated ad. Gaut says1 ‘indistinguishable from reality.’ Except for minor issues. The voice is clearly fake. The video in many places has the AI nature in its details. The English and the underlying ideas are both a kind of other-more-alien-80s-Japanese level of not right in a different direction in a way that no actual human would create. The lack of real video, anything beyond a simple single motion, is hard to miss once you notice it.
Are we making progress? Oh, very much so. I do not think the ‘last mile’ here will be so easy. Instead, we will have tools that make creating not-quite-right quality stuff much much faster, in a variety of ways, and also give people creating good things new tools that make them more productive as well.
Which, mostly, is great.
What it definitely isn’t is anything to panic about:
The ‘follow me for more AI news’ here is never unrelated to the rest. Keep in mind.
Oh, also:
New image model available, Deep-Floyd. Is it better than Stable Diffusion? I have no idea, anyone have a report?
Anime brings you Ghibli Diffusion.
Did ChatGPT Save Chris Munn Over $100k?
Mu. Check out the story SOP = Standard Operating Procedure.
The key question is: What is the counterfactual here without ChatGPT?
If the counterfactual is ‘oh well, I guess we won’t have an SOP for these things and we will have to write a new email every time a customer is 30 days overdue’ then ChatGPT is doing fantastically valuable work, great stuff.
If the counterfactual is ‘we would have had a human write the SOP for these things and have a human write the template emails’ then ChatGPT saved you a day of someone’s time? Maybe a few days? Humans are not as good as ChatGPT at quickly producing acceptable versions of such documents, they also aren’t that slow and their final results are at least as good.
The thing that is saving most of the money and time here is what Munn knew to do long before ChatGPT, documenting procedures, writing up the SOP, creating standard template communications. That’s standard automation stuff, not AI stuff.
The AI then assisted with figuring out how the tasks work, which is not so valuable exactly because we are going to create documentation. That means that rather than lots of people needing the AI to help them all the time, he only needed the AI to help him out once, then he was done. So not that much labor saved.
MedicGPT
Epic Systems and their medical records are partnering with Microsoft and OpenAI. The announced uses are drafting message responses to patients and discovering trends across patients, with those trends leading to both new scientific discoveries or financial opportunities. Drafting patient messages, and other letters and communications that need to be sent to various people, seems like an excellent use case, speeding up care provision. On pattern recognition? Less clear.
Or you could go for maximum ‘oh no please don’t do this’ and have a bot called ‘Dr. Gupta’ created by Martin Shkreli, I assure you I am not making this up.
Deepfaketown and Botpocalypse Soon
If you haven’t watched this yet, I don’t say this often, but: watch it (2:33).
Michael Shumacher’s family planning legal action against German weekly magazine that showed an AI-generated ‘interview’ with the Formula One champion.
Tom Brady threatens comedians behind AI standup video (NY Post). It would be a real shame if this caused lots of people to watch the 10 minute video.
Republican National Committee using deepfakes in anti-Biden ad, to depict the dystopian future where the incumbent president is re-elected. Quality level:
I don’t get why they didn’t use a real image here?
Plug-Ins as SuperPrompts
Brian Roemmele points out that when ChatGPT uses a plug-in, you can inspect it, and what do you find? A prompt, same as everything else. A ‘superprompt,’ he calls it.
Age of Massive Scale-Based AI Improvements Already Ending?
Such things are always in relative terms, things could go super duper fast while still being 10x slower than the last few months, yet Sam Altman thinks so? He is continuing to emphasize this narrative:
This is potentially your best possible situation. Throwing compute at bigger and bigger models and seeing rapid gains seemed like the easiest way to ensure we all get killed, so it is much better that this plan won’t work. Everything slows down. Pauses, even.
The question then is, what improvements can be found elsewhere?
There is a ton of space available for improvements. People haven’t even scratched the surface of what one can do to tune and take advantage of current-generation AI technology. Every time I think about how to do anything I get the urge to start coding or found a company. There’s a constant sense of ‘I figured out how to do it.’ Or at least, how to do it better. We’re going to see a bunch of bespoke detailed work.
Mostly, I think that’s great. I want to see maximum extraction of mundane utility in non-dangerous ways. In many ways I think the more bespoke your implementation of the things you want to do with current tech, the less danger you are posing, because a lot of what you are doing is de-risking the process, taking away ways for it to go wrong. You can still make things worse if you then use that to trust the AI too much.
It does seem like there is at least some ability to get better LLM performance by going larger, in exchange for more compute cost and more up-front training cost, and everyone is mostly going ‘nope, not worth it’ given the trade-offs involved, at least for the time being.
Eric Gilliam draws a parallel to innovation in Ju Jitsu, in the sense that OpenAI has already innovated a bunch, so one should bet on them innovating again even if they can’t rely on stacking more layers. It’s a consideration, at least.
It’s Time for the Merge
DeepMind and Google Brain have merged and are now a single unit, Google DeepMind, to be led by Demis Hassabis. The announcement is otherwise detail-free, leaving us to speculate.
Margaret Mitchel, founder and former co-head of Google’s AI ethics unit (she was fired, stated reason seems to be violations of security policies, there were also other issues related to the firing of Timnit Gebru) has a thread suggests that the Google Brain brand was damaged in terms of hiring and retention, which the thread heavily implies she blames on the firings. Sees this move as a way to fix that issue by moving the branding onto DeepMind.
Also she suggests this lets Google Brain founder Jeff Dean get away from managing people and back to writing code and building stuff as Google’s Chief Scientist, which is what Jeff is better at and wants to do anyway. There’s discussion about whether the involved ‘promotion’ is ‘fake’ and issues of internal politics, thing is that not everyone wants or would benefit from ‘real’ promotions as opposed to being able to do what they love doing and are good at.
My best guess is that this was pretty simple. Google Brain let its lunch get eaten by OpenAI and Microsoft, then was given one job. That job was called Bard. The initial Bard offering sucked and improvement wasn’t coming fast enough. So they called in DeepMind to fix the mess, put Demis in charge and convinced him to actually build and ship commercial things this time as part of the deal.
Europe’s Ongoing War on the Internet and All Human Activity Potentially Bears Dividends
If you are going to systematically make life worse with things like GPDR, make sure that every piece of data about anyone is proceeded by a cookie, disposed of exactly when required and also held onto and used in various ways, and require everyone who ever collects any bit of data on the internet justify their “legitimate interest” in doing so in every case, and generally tell everyone they’re not allowed to do anything?
If those regulations don’t actually make any sense in the context of actual human life?
Well, let’s also do it in the one place this might do some good, perhaps. AI.
MIT Technology Review’s Malissa Heikkilä writes “OpenAI’s hunger for data is coming back to bite it.”
Flagrant violations, you see, because OpenAI simply took data from the internet.
One does not simply take data from the internet, Europe retorts.
How dare they use publicly available information! They must be stopped.
Europe telling OpenAI not only to get out of the continent but that it must delete GPT-4, perhaps also GPT-3.5, and all the training data used for them would be quite the power move. Especially given they’re attached to Microsoft, who can’t exactly say ‘no.’
That would be much, much more impactful than any proposed pause. Why aren’t the Europeans worried we’ll ‘lose to China’? Why is Germany burning so much coal?
That’s the thing. OpenAI is doing the tech thing where they pretend that ‘where feasible’ has anything to do with such requirements. It doesn’t.
Then there’s the additional question the Italians are on about, which is how it collects users’ data during the post-training phase, things like sensitive chat logs.
Those problems seem eminently solvable, there’s no reason OpenAI can’t delete those chat logs on request or avoid sending along user data. That one, to me, is on OpenAI.
That is classic European regulator speak, of course you should document absolutely everything you ever do and everything should always be fully reversible and available to be deleted on anyone’s whim. It’s your fault for not planning ahead.
In any case, full compliance is clearly impossible, as Dustin Moskovitz observes.
It is not possible to train an LLM without any data that anyone might, in the future, ask to be removed, and still get something all that useful, even in theory. It would not know about any particular people, at all, or any events involving such people.
Gok is correct here, one way or another:
And, as an illustration of how such folks think, these were Paul’s only two replies:
Or Winfred Jebasingh, Gary Marcus and others saying it outright:
It seems Gary Marcus really does think that current existing LLMs should be banned?
How, exactly, would OpenAI have ‘taken of this from the beginning’? It seems pretty impossible.
Would it be possible, by working extensively over many years directly with such regulators, and spending tons of resources, and getting appreciably worse performance for it, for someone like Google or Microsoft to find ways to work with such regulations? Well, somewhat. The regulations still change. The regulations still at core are incompatible with reasonable operations. What happens in practice, when compliance is impossible? That is a negotiation, if you have a seat at the table.
I want to stop here to once again say to people like Tyler Cowen or Robin Hanson, yes. I get it. This is all completely insane, utterly ludicrous and nonsensical. It is standing in front of ordinary normal activity, and any attempt to make life better, and saying to it that You Shall Not Pass. It is, in almost all other realms, the enemy of all that is good.
You can think Brexit was a good idea. You can think Brexit was a bad idea. If you can’t understand some very good reasons why one would want to risk all to flee from the European regulatory agencies before it is too late, even if it might not work? That, my friend, is on you.
So the first thing to note is, the EU are doing things about this crazy all around you, it’s doing lots of real damage around the world and to the internet and to anyone’s ability anywhere to do anything, and others like the USA are not far behind. Imagine the world we could have if they stopped, or even reversed course.
Do I think AI capabilities development presents an existential risk to human life and all that I value, such that anything that I have at least mixed feelings about anything that promises to slow it down? Oh yes, very much so. But this is a coincidence, it has nothing whatsoever to do with what the EU and its member countries are doing here.
Paul Graham is even less charitable than I am here.
Except he fails to notice the answer, which is that you simply ban everything until you have banned AI. Since when do they worry about precision when banning things?
Contrast this to Chuck Schumer’s plan to require knowing ‘how the LLM came to its decision.’ Which is at least somewhat incompatible with how LLMs work, while also pointing to an actual problem that we don’t know how LLMs come to their decisions.
Not that this is only a European phenomenon, or one that America avoids. What’s a clear example? America is doing a similar thing with crypto, via the SEC. Take a bunch of regulations that make no sense in context, and that make the basic productive use cases impossible.
Even with slowing AI capabilities development being perhaps the most important thing, there’s still a super strong ‘not like this’ vibe. I mean. Not like this.
They Took Our Jobs
Will generative AI disproportionately impact women’s jobs over men’s? The Goldman Sacks analysis says yes, as reported by the Kenan Institute. As usual, ‘impacted’ does not tell you whether there will end up being more or less jobs, or whether jobs will have higher or lower pay, although the presumption is it is bad for employment.
The key question here might be whether generative AI is more fundamentally a complement to human interaction or whether it is a substitute for that interaction. Lots of other implications there as well. Similarly, what types of work will rise in demand in a world of generative AI, or have their latent demand deployed?
Another angle might be to ask where women have a comparative advantage versus where men have one, and ask what happens to those advantages with generative AI. I will intentionally leave it there.
A senior engineer asks what it would take for an AI to do their job, the answer comes back that the AI cannot do any of the actually important parts of the job and is not close to gaining such an ability.
This post succeeded at increasing my probability that LLMs being able to do high-level effective thinking and complex tasks is going to be hard and require many new insights. And that AutoGPTs are going to need superior core models before we have that much to worry about, in ways that clever scaffolding can’t fix.
The mental model here is something like: LLMs can do a lot of tasks and subtasks we do better than we can. Being able to automate and improve such tasks is a big productivity boost to many activities. Often combining those tasks and substasks will actually be mostly or entirely sufficient to solve a bigger problem, or even constitute the central thing about someone’s job.
The post also updated me towards thinking I could be quite a good engineer if I devoted the time to that, especially now in the GPT age, as the things the LLM is missing here are places where I should do well, whereas the places I am weak are increasingly not as crucial because they are a combination of fixable and outsourceable.
Study on Generative AI at Work finds customer service issues resolved per hour increased by 14%, with novice and low-skilled workers benefiting more and high-skill workers benefiting little. Customer sentiment was improved, requests for management intervention reduced, employee retention improved. Their interpretation is that AI allowed newer workers to catch up on knowledge and skills. In the context of customer service, it makes sense that advanced workers have already ‘maxed out’ and the remaining problems are the people calling for help – the ways in which one could optimize further are not available to them or an LLM. Whereas LLMs are very very good at customer-service-knowledge like tasks for those who don’t yet know.
Go Go Gadget AutoGPT
Roon speculates that the next barrier to AutoGPTs is not that the core model isn’t smart enough, it’s that it isn’t creative enough because its creativity has been intentionally destroyed by RLHF.
I suspect there is a lot to this. I think the future lies in using multiple distinct models as part of your bespoke AI agent or AutoGPT. Smaller fine-tuned models are cheaper, and better suited to many particular tasks. You want the ability to scale up or down the quality of your response, as needed – humans do this constantly, most of the time I am several numbers behind my maximum GPT-level and the same goes for you. One of those tasks with its own model could be the part where the AI needs to be creative, called whenever the program observes it is stuck or needs to generate options.
One Million Tokens
Paper: Scaling transformer to one million tokens and beyond with RMT.
I am not technically qualified to evaluate this paper.
Pete explains:
Well. You could try those things.
I’m going to go ahead and say absolutely not, it will not be able to write a novel without its hand being held every step of the way, at least not at a quality level anyone would want to read. A context window of that size is helpful, but if you can’t write a good 8k token story, how are you going to write a series of novels?
Could you chat with it for months? Sure, if you’re willing to pay what that costs. There are lots of great things one could do with a much, much larger context window. That goes double if you can get an LLM that hasn’t had all its creativity destroyed by RLHF.
In Other AI News
Overview from AI Supremacy of the major rivals to OpenAI. A big drop-off after DeepMind and Anthropic. No sign of China anywhere.
Snapchat rolls its ‘My AI’ feature out to everyone.
From Wired, Stack Overflow to seek payment from those training LLMs for the use of its data, since the creative commons license requires attribution of each individual question and answer back to community members, which LLMs cannot provide. Reddit plans to do the same, and Twitter as previously mentioned has mentioned plans to sue.
UK announces 100 million pound investment in AI capabilities and deployments.
I loved this line, wonderful British understatement:
ChatGPT offers to delete chats permanently after 30 days if you want that, and plans to in future offer ChatGPT Business on same terms as API data usage.
GPT-4 continues to have a sexism bias problem, it seems, under some reasonably central conditions.
More clarification that AI-generated images and text cannot be copyrighted. Could potentially be pretty great.
More Takes on China’s Draft Regulations
Helen Toner offers a thread.
For a skeptical take, here is Lao Mein assessing the Chinese AI safety community as essentially non-existent, the concepts are responded to with a ‘that’s interesting’ and then forgotten, and saying that all we’re seeing is a Chinese PR statement ‘saying the right words’ the same way they’d say ‘diversity’ or ‘inclusion.’ Warns not to bring any LW/EA style work into China, as this would only drive people away.
I see Lao as describing an entirely different mechanism than the one I am envisioning. Nothing in the Chinese statement talked about existential risk. I agree that China isn’t going to soon realize the existential dangers and want to stop for that reason. I am saying that China has other good self-interested reasons to want to perhaps slow things down.
Thus, while accepting the factual claims in Lao’s post as presumably accurate, I do not see them as changing my viewpoint here. I would be unsurprised if talks failed, but it seems crazy to say that talks are so doomed to fail that they should not be attempted, or that we should never go down any path that would involve future such talks.
The Quest for Sane Regulation
From Anna Lenhart, A Google Doc lists Federal Legislative Proposals Pertaining to Generative AI. Almost no one should respond by reading the list of proposals, although some people I know really should do that. For most people, the point here is that there really are a lot of proposals. A lot of it is about privacy concerns.
There was a Senate Committee on Armed Services hearing on AI, in which the CEO of Palantir urged us to spend 5% of our budget on capabilities that would ‘terrify’ our enemies, and that we need to hook our weapons up to our AI systems as soon as possible, or else ‘China will surpass us in a decade.’
You see, we need ‘Democratic AI’ or else China will build ‘socialist AI,’ oh no. And by ‘Democratic AI’ we mean United States military AI.
Much time was spent on China. The pause was treated as you would expect military types to treat it, e.g. Senator Mike Rounds (R-SD) saying:
Well, yes, obviously, if you are looking at things purely from an adversarial ‘US security’ standpoint and don’t understand existential risks, then not building up capabilities is the greater ‘risk’ regardless of the underlying facts. No matter what, the answer will always be ‘the US military comes first’ because you made that assumption up front.
The headline of the linked piece was ‘Even The Politicians Thought the Open Letter Made No Sense In The Senate Hearing on AI’ which shows how much those who want to jam the accelerator on capabilities treat arguments as soldiers. Why would politicians focused on competition with China be an ‘even’ here? Why would you use the term ‘made no sense?’ Exactly.
Also: One of the standard justifications people give to ‘the AIs are nothing to worry about’ is that we would never be so stupid as to hook up all our weapon systems to AIs.
I hope by now it is clear to everyone that all talk of not hooking our AIs up to our weapon systems is Obvious Nonsense.
And yet, what is actually happening here? Mainly calls for regulation and restriction.
Yes, these restrictions are mainly aimed at China, as one would expect when the goal is ‘beat China.’ It’s still a start, as are the ‘certain guarantees of security.’ Which, if actually taken seriously, could well prove rather impossible to satisfy.
Pascal’s Mugging Mugging
Here is some of a failure to communicate, that really shouldn’t be happening:
Many such cases.
The whole idea of Pascal’s Mugging is that if someone presents you with evidence that gives you a very very low (not zero!) probability of a very very extreme result, then this can cause that extreme result to dominate your thinking, and it is a mistake to let this happen. This was originally meant to apply to probabilities like ‘1 in 10^10.’ A weaker form of this is still a non-trivial concern at numbers like 1% in some social contexts, but this doesn’t apply at all by 5% unless you are insufficiently calibrated to be able to distinguish 1% from 5%.
There are a remarkably large number of people who have interpreted Pascal’s Mugging to mean various very different, highly non-sensical things.
In particular, saying ‘if you do X then we all die with probability 99% so don’t do X’ is the opposite of Pascal’s Mugging. It’s not saying that a small probability should dominate your thinking, it’s saying that a very large probability of a very large impact should dominate your thinking.
If you don’t think that a very large probability of a very large impact should dominate your thinking, and think this is a ‘dishonest argument technique’ then I notice I am confused? Presumably you would think that what Perry meant to say is that he thinks that the probability claims here are not reasonable, which is a reasonable objection to make. That’s a completely different claim. But no, if you click through you see Perry doubling down, saying that exactly because he thinks the probability estimates are wrong, that this means it is a Pascal’s Mugging. Or that other people might make claims that smaller probabilities would justify worry, again making it Pascal’s Mugging. Or that Eliezer wants to ‘risk nuclear war’ so Pascal’s Mugging again.
Rohit’s statement is even weirder and more confused, claiming that a call to not do the thing that almost certainly kills you is Pascal’s Mugging, because it ‘concentrates on the small probability of a good outcome’? This actually flat out is a failure to parse, unless it is something far worse. Yet such reactions are common.
My presumption is the actual logic is something like:
That sounds harsh, yet it is what fits the facts.
To Foom or Not to Foom
A key question for many is, how long will it take to get from AGI to superintelligence? Will it happen in days or even hours, the pure classic ‘foom’ scenario? Will it happen in weeks or months, a slower-motion foom that counts in some ways and not others? Or will it take years or decades, or perhaps never happen at all?
Opinions on the probability of a classic foom run all the way from Can’t Happen, to it being the baseline scenario but highly uncertain (this is my understanding of Yudkowsky’s position, and is also mine), to others who think of foom as being almost inevitable once we hit AGI.
What do predictors think? Metaculus says ten and a half months.
The number has bounced around a bunch, from a low of about four months to as many as two and a half years.
They also have ‘the first general AI system devised, tested and publicly announced’ scheduled for June 2032.
It’s a cool little prediction question, even if it can never be meaningfully graded. The prediction seems reasonable.
It is also fully compatible with a wide variety of levels of risk of doom. If we get a true foom in hours or days, most people I talk to agree the risk of doom in that scenario is very high. If we get a slow-motion foom, the outcome is non-obvious. Those who have a very high probability of doom mostly have that probability not because they think full-speed foom is inevitable, but because they think there is little we would actually do, in practice, within those few months that would matter.
On a fun related note, Rohit asks:
To which I would reply, this is AI-complete, at which point the AI would solve the problem by taking control of the future. That’s way easier than actually solving the Socialist Calculation Debate.
Or: The only reliable way to centrally set the right price of coffee is if everyone’s dead.
Tyler Cowen links to Perry Metzger arguing that it is impossible to foom, because there is nothing it could possibly do that wouldn’t take months, since it would start out only slightly above human level? Combined with arguing that the only way to be smarter than human is to use trillions of parameters or even hundreds of trillions, because nothing else is that helpful?
To give you an idea, here’s the first 4 Tweets and some key later ones, click through if you want the whole thing.
I do appreciate the attempt to actually think about such questions. And yet, sigh.
Yes, doing new things is hard if you presume that:
So yes, if you rule out the idea that the new AI system could figure out anything humans can’t figure out, or do anything humans couldn’t do, or do anything importantly more efficiently than we could, then it’s not going to foom.
I can’t argue with that, except to point out it is assuming its conclusion. Any foom would require that the thing that goes foom, before going foom, is able to access fundamental insights that Perry’s model excludes as impossible – or at least, impossible to find quickly without first going foom.
There is some chance we live in a physical universe that is close to Perry’s view. In that universe, a true hours-to-days foom is physically impossible, and a super-fast smarter-than-human-but-not-that-much-smarter system has no insufficient low hanging fruit that it can pick quickly, so it has to go slow.
Maybe yes, maybe no. My guess is no, but I’d still say maybe 30% chance?
If we do live in such a world, I would still note two things.
One, Perry admits, in this thread, that such a system could potentially get as far as taking over a wide variety of online resources, and in a longer time period do all the things that would constitute a foom. It wouldn’t fully play in hours, instead more like months or years, it is still the same playbook ending in things like superintelligence and nanotechnology.
So, in that scenario, suppose we don’t get alignment right on this first try. The system that is in what one might call a ‘slow motion foom’ makes, as Perry suggests, many clones of itself, gains access to lots of computer hardware and economic resources (likely while making itself very helpful!) and through experimentation and iteration becoming more capable. It hogs sufficiently many of Earth’s available relevant resources to bottleneck and slow down any potential rivals. Then it does its thing, whatever its thing might be, after a few years.
At what point, and how, are we going to realize what is about to happen to us, and stop this from happening? What good does all of this buy you? The question that matters is not whether we all die or see a giant foom within a few hours. The key question is when exactly the process becomes, in practice, given our Don’t-Look-Up world, something humans would be unable or unwilling to stop. Keeping in mind they don’t know that they need to stop it.
Second, Perry’s model includes an important core assumption: No one ever gets anything like this right on the first try.
Uh no.
In other Metzger news, Eliezer Yudkowsky offers, in this thread, various commentary on various Perry Metzger statements. I confirm that Perry has been very aggressive about asserting what Eliezer claims or believes, often in ways that do not match either my understanding of Eliezer’s claims and beliefs, and that do not match Eliezer’s own statements.
Back to the question of the feasibility of foom, Jacob Buckman argues it will require at least one paradigm-changing breakthrough.
This seems like a clean, well presented form of the argument we encounter a lot, that says something like:
Jacob seems to be thinking well about these problems. If an AI is to foom, it is going to have to figure out how to do DRL that works, or figure out how to generate its own high quality data sets, or figure out something new we haven’t thought of that does the job, some innovation we’re not thinking about. Right now, it doesn’t have the capability to figure such a thing out on its own.
I notice this framing does not make me so confident it won’t happen soon? If anything, this seems like it is flashing neon lights of ‘here’s how you would do it, over here, although I don’t know how to implement it yet.’
The extension of the argument, as I understand it, is something like: Machine learning, like most science at this point, not about that One Weird Trick or clean fundamental insight, it’s about lots of tinkering and trying things and incremental improvements. Even if either an AI system or a human hit upon the next great important insight, that wouldn’t help much until there was a ton of tinkering done, which would be expensive and take a lot of time.
My worry is two fold there. First, I am not convinced it is fully true. Second, I am not convinced that this changes the answer all that much, in the sense that what still happens is that once you’ve done enough tinkering the resulting system still eventually crosses some threshold where it can take over the process itself instead of being unable to do that, and you still get a phase change, and also that quite plausibly comes with a bunch of other gains that greatly speed up what happens next.
I also had a bunch of thoughts of ‘oh, well, that’s easy, obviously you would just [OH MY LORD IS THIS CENSORED]’ and I’m presuming all of them have been tried many times and found stupid and non-functional given my lack of detailed domain knowledge and how many people work on the problems, but still I’m going to leave it to your imagination.
Robin Hanson Proposes Requiring Buying Insurance
In this case, he wants people to be required to buy foom insurance.
He proposes such insurance a lot, as the solution to many of society’s problems. There is always a good incentive story to be told in theory about such requirements, until one thinks about them briefly to ask how it would work in practice – I’ve had to write ‘do not explain why Hanson’s latest required insurance proposal wouldn’t work’ on my mental blackboard a few times. Often these are still good ideas to move towards, on the margin.
Foom liability, and requiring foom liability insurance, is of course hilarious due to its impossibility, a classic case of We All Will Go Together When We Go.
Robin says that’s fine, actual successful foom isn’t the problem because of course first we will see failed attempts, for which the insurance can pay out. So we can impose strict liability and punitive damage in such cases, and that will provide good incentives. Of course, if you were sufficiently confident Robin was right about this, you wouldn’t as much need his intervention, although it still would be helpful.
I agree with Eliezer that, despite the premise and initial argument being a massive misstatement of the requirements for foom in the aims of dismissing such worries out of hand without engaging with the actual dangers at all, such liability and insurance requirements would indeed be good ideas that could actually help. Getting the kind of conservatism you see in insurance companies involved and adding such risks to the bill seems great.
It would be remiss not to address the massive misstatement, which he also repeats on the Bankless podcast among other places. He says,
Eliezer responds:
On several of these particular points, Eliezer is so clearly correct that I am confused how Robin is making the mistakes in question. In particular, for these two Eliezer mentions: You don’t need to explicitly task an AI to do self-improvement in order for its best move to be to self-improve, so point (a) is flat out not required. Its values do not need to change, even in the central case, because the central case is that the AI’s values were not well-defined in the first place, at least not towards anything you would like, saying (g) is required is not reasonable.
There’s a strange confusion here in terms of the advance being secret – there are a bunch of different meanings here, both secret to the lab training the AI, and secret to those outside the lab, and also the secret that some such method exists versus the secret required to do the method. How much of this secrecy is actually needed? For the foom to be single-source, either (b) or (h) style hidden info is sufficient, as would be even a small head start if growth was sufficiently rapid, or some labs (quite correctly) are hesitant to deploy while another charges ahead. Then there’s the question of ‘does the lab in question notice in time’ and yes by definition that is a step, with the caveat that we have quite a lot of evidence that they probably won’t act in time.
And also the whole framing of ‘foom only happens in exactly this scenario’ and ‘this requires all these stages of doom’ the way Nate Silver described Donald Trump needing to survive six nomination steps to win, is something I increasingly have trouble interpreting as a good faith understanding from those who should know better by now, even before the ‘if no foom then nothing bad ever happens’ follow-up line of logic.
It’s more than a little like saying ‘oh, so you’re saying for Einstein’s theory to be true you have to not be able to travel faster than light, and there has to be this particular exact conversion rate between matter and energy, and space has to bend in exactly this way, we’d need to have this exact wobble in this planet’s orbit, and…’
Quiet Speculations
DeepMind paper proposes using Rawls’ Veil of Ignorance as AI governance principle, finds people endorse helping the least well off more when the veil is applied. While I do appreciate that anyone is trying to do any philosophy or problem solving at all on such questions: Oh no. Please do not bring Rawls into this. Please do not pretend that the primary point of the universe is ‘justice.’ Please do not endorse the judgments of risk averse confused people under extreme uncertainty to sacrifice efficiency and all future growth prospects for moment-in-time distributional fairness. Do not want. It is crazy to me that Rawls continues to be taken seriously.
The Economist asks: Rapid progress in AI is arousing fear as well as excitement. How worried should you be?
Quintin Pope says that ‘evolution provides no evidence for the sharp left turn,’ which is the predicted phenomenon where an AI system would act friendly and as we would like it to, until it gets to a threshold level of capabilities and power where it can successfully turn on us, at which point it would turn on us. Mostly Quintin is saying that evolution is a bad analogy for AI development, and that the human ‘sharp left turn’ against reproductive fitness towards other things happened because culture, which allowed transmission of learning across generations. Whereas current systems effectively, in this metaphor, ‘already have culture.’
I am confused why this brings any comfort. This seems like an argument that the sharp left turn is impossible without something that the AI already possesses. So if anything, it is arguing that rather than our AIs suddenly being misaligned in the future, perhaps they are already misaligned now? I don’t see how that helps here? What this is saying, correctly I think, is that we already have the issue that our AIs are not well aligned and will do instrumentally convergent things and perversely optimize their utility functions and that may well involve turning on us if and only if that plan would work.
It might still be an argument against a sharp left turn in the sense that people were making a stronger argument about evolutionary implications than this.
Whereas to me the ‘evolutionary argument for the sharp left turn’ is that humans exhibit sharp left turns constantly. I don’t mean in the sense that humans suddenly care a lot about things other than their genetic legacies. I mean in the sense that humans will often wait until they have sufficient power to get away with it, then turn on each other. This is common, regular, very ordinary human strategic behavior.
The sharp left turn… isn’t… weird? It’s… very standard?
Speculation that China might ‘win the race to GPT-5.’ Not because we’d pause, simply because China is ‘more innovative’ and cares about ‘sovereign AI’ and the country has deep pockets. Has a quote from Microsoft president warning about the Beijing Academy of Artificial Intelligence. It’s pretty amazing to me how much people can talk themselves into worrying about China. I would bet rather heavily, giving rather good odds, that the first ‘GPT-5-worthy’ system comes from America.
A very Slate pitch, in Slate, calls for an ‘AI public option.’ You see, there’s ‘good reason to fear that A.I. systems like ChatGPT and GPT-4 will harm democracy.’ So instead we need an LLM built ‘for public benefit.’ I do not believe anyone involved has thought this through, or understands what goes into creating an LLM. Robin Hanson asks ‘how would we trust it?’ and that doesn’t even scratch the surface of the issues here.
The Doom Memeplex
Chana Messinger notes Jan Kulveit’s observation that the ‘doom memeplex’ is potentially unhelpful in solving the problems currently dooming us. Like any good bureaucracy, the doom memeplex has a symbiotic relationship with actual AGI doom and the race to build an AGI system that would doom us, rather than benefiting from solving the source of doom.
To what extent is a memeplex going to be able to take control of people’s beliefs and statements? This does clearly happen sometimes, resulting in death spirals of ever-increasing extremity. I believe that social information cascades definitely contributed, at various points in the past, to excessively high-doom estimates happening in Berkeley.
I do not think that is happening now for MIRI, and I hope not for others either. I strongly agree with Rob that there is a strong social disincentive to extreme doom estimates.
Estimating a 50% chance of doom is ‘good for business’ in the sense that this is where one’s impact is at maximum. The world hangs in the balance, your decision can make the difference. One can argue that something like 80% is strategically superior, if what is needed is a dramatic course correction, the ability to pull off something extremely difficult, and perhaps push that to 90%.
Whereas 99%+ estimates of doom are clearly very much strategically and socially anti-useful when you don’t believe them. You risk making people despair and not act. You risk being dismissed as a crackpot or unreasonable, or unwilling to consider that you might be wrong – at minimum you seem super overconfident. Why not pull back a little?
Two reasons. One, the value of speaking truth, of giving your actual real probability estimate. Two, if things being so dire changes the correct actions to take, that one must do things that ‘buy more dignity’ rather than trying to assume a hopeful situation and easy problem. The thing is that both of these depend on you believing your answer, for real. Otherwise, it’s no use.
People Are Worried About AI Killing Everyone
Max Tegmark gives the Time magazine article version of his case for taking seriously the fact that AI is on track to kill everyone. I found this to be an excellent piece, if you want a straightforward no-prior-knowledge-needed place to point someone regarding existential risk from AI this is a pretty good choice.
Michael Nielsen thread collecting essentially all the founders of the three major AI labs (DeepMind, OpenAI and Anthropic) expressing explicit concern that AI might kill everyone. The concerns seem expressed with varying degrees of seriousness, sincerity and willingness to do anything about them. Talk is always cheap.
Let us define an unsolution: A proposed solution that not only fails to solve the problem, but also by being proposed makes the problem worse or creates new problems.
Paul Christiano, who among many other things invented OpenAI’s RLHF technique, says 50% chance of doom once systems are human-level intelligent, the good news is only 10%-20% of that is full AI takeover, the remaining 30%-40% is that we die in some other way shortly after the human-level AI milestone, such as the use of some of the resulting destructive technologies.
Elon Musk continues to be worried. Continues to propose unsolutions.
I think Roon is closer to correct here, yes if Elon actually understood the problem Eliezer would be right, but Elon doesn’t, which explains why his solutions continuously do not make sense. So Rob’s reply is most helpful.
Elon also worries about other aspects, like weapons.
If anything, reports of people explicitly and directly hooking AIs up to weapon systems and having it direct battles makes me feel better, because it potentially wakes people up to what is inevitably going to happen anyway. At least this way we can notice and it can be common knowledge, instead of ‘easy, we simply never hook the tech up to the nuclear weapon systems’ or what not, yes, we are exactly this stupid and undignified, please stop pretending we are not and save us all some time and typing.
Elon met with Chuck Schumer about AI regulation. I do not actually think we know which regulatory proposals Elon thinks are good? He is very hard to anticipate in such things, you can’t work it out by logic.
Technically Darren Jones MP (member of UK parliament) is not quite there, this is still a quite reasonable way to start.
Once you get that you are creating something which could be dangerous while you are testing it you’ve made strong progress towards realizing the real problems. Also, yes, see how you can actually suggest trying for an international framework?
Couldn’t be.
I always see ‘you want X here in Y context? But you usually oppose X-like things!’ as an argument that the person isn’t allowed to endorse X in Y context. There can be an argument there, but more commonly and importantly when you see someone who is strongly anti-X in most contexts make a clear exception, that is a very strong statement that there is an unusually strong case for X in context Y, relative to other contexts.
Other People Are Not Worried About AI Killing Everyone
Ben Goertzel argues, after launching some ad hominem attacks, that LLMs like GPT-4 will never be AGIs – “they utterly lack the sort of cognitive architecture needed to support human-level AGI” – and that they’re entirely unimpressive, therefore they should be open sourced.
Sure, that makes sense if we were confident LLMs could not be or lead soon to AGI, that AGI was far. So, does that mean he would propose slowing down if AGI was near? Oh, no. Quite the opposite.
Instead, he says, we should have an actually open effort to create real AGIs as quickly as possible, because innovation is good and humans should always keep innovating. And that we should treat LLMs as potentially crucial cognitive infrastructure components to future AGIs.
He then says, we need not worry about the fact that most intelligent minds won’t care about the things we care about, because whatever AGIs we do create will be created using ‘cognitive architectures modeled to some extent on our own, and we are then teaching, training and raising these systems in our own culture and society.’
So Ben calls instead for decentralization, of ensuring that no one has the power to slow down or control future AGIs that we develop as quickly as possible, and trusting that the resulting competitions will go well for us, because of this… I don’t know what?
So, we’ll be saved by The Power of Love? As long as we keep the faith?
Dog, his house on fire, drinking coffee.
Or: As I dissolve: Any sort of Singularity is going to feel weird and complex with hard-to-analyze pluses and minuses.
I’m the weirdo. I want to think about the actual pluses and minuses and consequences of our actions even when this might conflict with the narrative or the vibe.
Autism Capital is too busy worrying about FTX, or rationalists, or EAs.
A claim that superforecastersare not worried about AI killing everyone.
While 0.38% is still crazy high compared to what I’d like that number to be, it’s crazy low compared to any reasonable estimate, and if I believed it was this low without my intervention, I’d have other things I’d worry about more than AI x-risk.
Several people noticed this doesn’t actually make any sense, unless the superforcasters in question were simply responding to incentives (as in, if AI does kill everyone, it matters little that you predicted that it wouldn’t do that).
It also does not make sense because the forecast flat out does not make any sense. People who put a <1% chance on this outcome by 2100 are not going to be super at forecasting. Or, alternatively, they are going to be good at forecasting when and to the extent that you provide them with good incentives.
If you have 100 superforecasters, and they give you very different probability estimates, do you take the median probability they give you, or do you take the mean probability they give you? If you think the superforecasters are actually good at what they do, I think you pretty clearly use the mean, which is going to be a lot higher.
The real question remains, how the hell did this happen? And what happened exactly?
If months of debate with superforecasters didn’t accomplish much, that’s really disheartening on the prospects of persuading others with arguments.
At the New Yorker, Jaron Lanier joins Team Stochastic Parrot, saying ‘there is no A.I.’ and calling for something called ‘data dignity,’ which I believe means tracking the data underlying any outputs, so everything is clearly labeled with where it comes from. It is all deeply founded on the idea that ‘AIs’ are and will remain simply remixers of input data, and mere tools. Yet still Jaron sees much danger.
The Lighter Side
Robert Wiblin: Wee did it everyone we proved that fire is safe.
1
Yes I am aware he is joking throughout.