Anyone want to help out? I have some ideas I’d like to try at some point.
I can help, let me know what are those ideas you have mind...
My view: People betting NO in this market are betting on either fizzle or doom. Hard for me to see how AI going well doesn't get here.
Sam Altman: is there a word for feeling nostalgic for the time period you’re living through at the time you’re living it?
Call it "nowstalgia."
You can’t collect your pension if the state declares you dead, and an AI in India is going around doing that, sometimes to people still alive. They say AI but I’m not sure this is actually AI at all, sounds more like a database?
Sounds like a new excuse for an old problem.
What good is it to buy fake followers, will people actually get meaningfully fooled?
I think this is a problem with advertising. On one hand, there seems to be a consensus that advertising is important. On the other hand, there are no specific predictions.
If you design a better mousetrap and buy ads on Facebook, how much you should spend on ads, and how many extra mousetraps should you expect to sell? No one knows. Heck, you can't even verify whether Facebook actually showed your ad to as many people as they said they would! If the sales are bad, are the mousetraps bad, or was your Facebook ad bad, or is advertising on Facebook bad and you should have advertised somewhere else instead? No one knows. (In theory, there are ways how you could attempt to measure something, but in practice, there is a lot of noise.)
So people just guess, and pray for a lucky outcome. They probably won't even know if they actually got it, because sometimes you have no idea how many mousetraps you would sell in the alternative reality without ads.
And that is the kind of environment where superstition rules; where saying "I have a million followers" impresses people, even if they suspect than many of them are bots. Still better than paying someone who only had thousand followers (especially considering that some of those may be bots, too).
In a week with two podcasts I covered extensively, I was happy that there was little other news.
That is, until right before press time, when Google rebranded Bard to Gemini, released an app for that, and offered a premium subscription ($20/month) for Gemini Ultra.
Gemini Ultra is Here
I have had the honor and opportunity to check out Gemini Advanced before its release.
The base model seems to be better than GPT-4. It seems excellent for code, for explanations and answering questions about facts or how things work, for generic displays of intelligence, for telling you how to do something. Hitting the Google icon to have it look for sources is great.
In general, if you want to be a power user, if you want to push the envelope in various ways, Gemini is not going to make it easy on you. However, if you want to be a normal user, doing the baseline things that I or others most often find most useful, and you are fine with what Google ‘wants’ you to be doing? Then it seems great.
The biggest issue is that Gemini can be conservative with its refusals. It is graceful, but it will still often not give you what you wanted. There is a habit of telling you how to do something, when you wanted Gemini to go ahead and do it. Trying to get an estimation or probability of any kind can be extremely difficult, and that is a large chunk of what I often want. If the model is not sure, it will say it is not sure and good luck getting it to guess, even when it knows far more than you. This is the ‘doctor, is this a 1%, 10%, 50%, 90% or 99% chance?’ situation, where they say ‘it could be cancer’ and they won’t give you anything beyond that. I’ve learned to ask such questions elsewhere.
There are also various features in ChatGPT, like GPTs and custom instructions and playground settings, that are absent. Here I do not know what Google will decide to do.
I expect this to continue to be the balance. Gemini likely remains relatively locked down and harder to customize or push the envelope with, but very good at normal cases, at least until OpenAI releases GPT-5, then who knows.
There are various other features where there is room for improvement. Knowledge of the present I found impossible to predict, sometimes it knew things and it was great, other times it did not. The Gemini Extensions are great when they work and it would be great to get more of them, but are finicky and made several mistakes, and we only get these five for now. The image generation is limited to 512×512 (and is unaware that it has this restriction). There are situations in which your clear intent is ‘please do or figure out X for me’ and instead it tells you how to do or figure out X yourself. There are a bunch of query types that could use more hard-coding (or fine-tuning) to get them right, given how often I assume they will come up. And so on.
While there is still lots of room for improvement and the restrictions can frustrate, Gemini Advanced has become my default LLM to use over ChatGPT for most queries. I plan on subscribing to both Gemini and ChatGPT. I am not sure which I would pick if I had to choose.
Table of Contents
Don’t miss the Dwarkesh Patel interview with Tyler Cowen. You may or may not wish to miss the debate between Based Beff Jezos and Connor Leahy.
Language Models Offer Mundane Utility
Paul Graham uses ChatGPT and Google in parallel, finds that mostly what he wants are answers and for that ChatGPT is usually better.
I think it is trying to compete? Although it is indeed a really promising technology. Also it is not eating half of Google’s business, although LLMs likely will eventually do so in all their forms. ChatGPT use compared to search remains miniscule for most people. Whereas yes, if I would have done a Google search before, I’m now about 50% to turn to an LLM.
Good news?
I have not been asking for code so I haven’t experienced any of the laziness.
Recover the text from Roman mostly very much non-intact scrolls from Pompeii.
Extract the title when using ‘Send to Kindle’ and automatically come up with a good cover picture. More apps need an option to enter your API key so they can integrate such features, but of course they would also need to be ready to set up the queries and use the responses.
Reshape ornithology.
Better answers from GPT-4 if you offer a bribe, best amounts are $20 or (even better) over $100,000. If you’re willing to be a lying liar, of course.
OpenAI offers endpoint-specific API keys, a big security win. A commentor asks why we can’t control the spending on a key. That seems like an easy win as well.
A 270M parameter transformer can play chess without search at blitz Elo 2895 via distillation, outperforming AlphaZero’s policy and value networks if you exclude all search, model of course is by DeepMind. It uses 10 million games with action values annotated by Stockfish 16, and nothing else.
Language Models Don’t Offer Mundane Utility
You can’t collect your pension if the state declares you dead, and an AI in India is going around doing that, sometimes to people still alive. They say AI but I’m not sure this is actually AI at all, sounds more like a database?
It is also unclear to me from the post what is causing this massive error rate. My presumption is that there are people in local government that are trying hard to get people off the rolls, rather than this being an AI issue.
Train an LLM as you would train an employee? Gary Tan links to discussions (and suggests using r/LocalLlama), context window limitations are coming into play and ruining everyone’s fun, people are trying to find ways around that. There are a bunch of startups in the replies pitching solutions. My inner builder has tons of ideas on how to try and make this work, if I had the bandwidth for an attempt (while I’d be learning as I go). If a VC wants to fund my startup and a high enough valuation to make it work I’ll hire software engineering to try a variety of stuff, but I do not expect this.
GPT-4 Real This Time
What is the latest on LLM political preferences in base models? David Rozado takes a crack. While he finds the traditional left-libertarian bias in deployed versions of LLMs, base models get a different answer, and are exactly in the center.
One way of thinking about this is that ‘what we want to hear’ as judged by those doing the RLHF training is reliably left-libertarian. No matter what you (if you are say Elon Musk) might want, in practice that is what you get. However, if you actively want RightWingGPT or LeftWingGPT, they are easy to create, so here you go.
OpenAI is working on an agent that will ‘essentially take over a consumer’s device.’
This was always coming, this speeds up the expected timeline a bit.
Colin Fraser’s note is apt here. The old OpenAI philosophy was incompatible with hype about future abilities that would doubtless drive others to invest more into the AGI race. The new OpenAI seems not to care about that. Nor does it seem to be that worried about all the risk concerns.
Reminder for those trying AutoGPTs of various sorts, if the model output is executed directly by the system, you are putting your system and everything that system can access at risk. Do not put into play anything you are unwilling to lose, and be very careful with what inputs the system is reading in what form. At a bare minimum, wait for the red teamers to give their full reports.
It will almost certainly not be that easy for an attacker, but the underlying problems continue to have no known solutions.
Fun with Image Generation
Copilot’s version of DALLE-3 now lets you edit images directly, at least among a fixed set of options.
YouTube’s annual letter says they plan to use AI to enable creatives, but everything discussed seems tiny and lame.
Deepfaketown and Botpocalypse Soon
Finance worker pays out $25 million after video call with deepfake ‘CFO.’ The worker had suspicions, but paid out because he recognized the participants in the call, it is amazing how often even when it works such schemes cause people to be highly suspicious. Obviously more like this is coming, and audio or even video evidence is going to stop being something you would rely on to send out $25 million. Some justified initial skepticism but at this point I presume it was real.
Oh, no! Looks like Bard will give you 512×512 images and they will happily produce a picture of Mario if you ask for a videogame plumber. So, yes, the internet is full of pictures of Mario, and it is going to learn about Mario and other popular characters. I am shocked, shocked that there are copyrighted characters being generated in this establishment.
DALLE-3 will now put metadata in its images saying they are machine generated.
Freddie DeBoer points out we have no ability to stop deepfakes. Yes, well. Although we can substantially slow down distribution in practice, that’s where it ends.
In a surprise to (I hope) no one, one of the uses that cannot be stopped is the Fake ID. It seems there is an underground website called OnlyFake (great name!) using AI to create fake IDs in minutes for $15, and they are good enough to (for example) fool the cryptocurrency exchange OKX. The actual mystery is why ID technology has held up as well as it has so far.
Davidad on threats to democracy:
Why is the ability to say different things to different people a ‘threat to democracy’? I do get that such things are different at scale, and I get that this might increase ad revenue, but it is a level playing field. It is not obviously more or less symmetric or asymmetric than untargeted ads, and offers the potential to offer more sophisticated arguments, and leave people more informed.
The ‘strategic falsification of political popularities’ also seems an add concern. There are very easy ways to check, via polls, if such popularity is real or not, and ‘draw attention to someone or some cause’ is a known technology. Again, I get the idea, that if you can swarm social media with bots then you can give off a false impression far easier, but this is already not difficult and people will quickly learn not to trust a bunch of accounts that lack human grounding and history. I am again not worried.
The falsification of audio and video evidence also seems not that big a deal to me right now, because as we have seen repeatedly, the demand is for low-quality fakes, not high-quality fakes. People who are inclined to believe lies already believe them, those who are not can still spot the fakes or spot others spotting them, although yes it makes things modestly harder. I predict that the worries about this are overblown in terms of the 2024 election, although I can imagine a bunch of issues with faked claims of election fraud.
What is the main threat to democracy from AI? To me it is not the threat of misuse of current affordances by humans to manipulate opinion. That is the kind of threat we know how to handle. We should instead worry about future technologies that threaten us more generally, and also happen to threaten democracy because of it. So the actual existential risks, or massive economic disruptions, transformations and redistributions. Or, ironically, politicians who might decide to move forward with AI in the wake of the public’s demand to stop, and who decide, with or without the help of the AIs and those working on them, to elect a new public, or perhaps they are forced into doing so. That sort of thing.
We have come full circle, now they are taking adult stars and adding on fake clothes?
They Took Our Jobs
Washington Post editorial asserts that AI is the true threat to journalism, that we must stop dastardly LLMs building off of other work with little or no compensation, warning that the ‘new Clippy’ will tell everyone the news of the day. I suppose the news of the day should be closely guarded? But yes, at least if the question is provision of very recent information, then you can make a case that there is a direct threat to the business. If ChatGPT is summarizing today’s New York Times articles rather than linking to them, or repeating them verbatim, then we do have an issue if it goes too far. This is very much not the situation in the lawsuit.
Paper says that LLMs are superior to human lawyers in contract review even before the 99.97% lower price. LLMs make mistakes, but humans made more mistakes. In the comments, lawyer Michael Thomas welcomes this, as contract review is very much a computer’s type of job. Everyone constantly predicts that legal barriers will be thrown up to prevent such efficiency gains, but so far we keep not doing that.
Get Involved
It doesn’t have to be AI! You got to give them hope. Sam Altman links to this list of ten medical technologies that won’t exist in five years, but that perhaps could, although given how we regulate things that timeline sounds like ‘good f****** luck.’ Of course we should do it all anyway. It is an excellent sign to see Altman promoting such things, and he does walk the walk too to a real extent. I agree, these are excellent projects, we should get on them. Also there are only so many people out there capable of this level of funding, so one should not look askance at those who aim lower.
MIRI still looking for an operations generalist.
Introducing
Nomic, an actually open source AI, as in you have access to the whole thing. No, it does not meaningfully ‘beat OpenAI.’
In Other AI News
Alibaba and Tencent fall off list of world’s ten most valuable companies as Chinese stock market continues to tank. If you are worried we are in danger of ‘losing to China’ there are many ways to check on this. One is to look at the models and progress in AI directly. Another is to look at the market.
Many OpenAI investors including Founders Fund, Sequoia and Khosla passing on current round due to a mix of valuation and corporate structure concerns, and worry about competition from the likes of Google and Amazon. In purely expected value terms I believe passing here is a mistake. Of course, OpenAI can and should price this round such that many investors take a pass, if others are still on board. Why not get the maximum?
US AI Safety Institute announces leadership team. Elizabeth Kelly to lead the Institute as Director & Elham Tabassi to serve as Chief Technology Officer.
Geoffrey Irving joins the UK AI Safety Institute as Research Director, Ian Hogarth offers a third progress report. They are still hiring.
Three minutes is enough for an IQ test for humans that is supposedly pretty accurate. What does this say about how easy it should be to measure the intelligence of an LLM?
British government commits over 130 million additional pounds to AI, bringing total over 230 million. It breaks down to 10 million for regulators, 2 million for the Arts and Humanities Research Council, then here are the two big ones:
As usual, ‘invest in AI’ can mean investing in safety, or it can mean investing in capabilities and deployment, which can either be to capture mundane utility or to advance the frontier. It sure sounds like this round is mostly capabilities, but also that it focuses on capturing mundane utility in places that are clearly good, with a focus on healthcare and science.
Smaug-72B is the new strongest LLM with open model weights… on benchmarks. This is by the startup Abacus AI, fine tuning on Qwen-72B. I continue to presume that if you are advertising how good you are on benchmarks, that this means you gamed the benchmarks, and of course you can keep fine-tuning to be slightly better on benchmarks, congratulations everyone, doesn’t mean your model has any practical use.
Need is a strong word. Demand is the correct term here.
There will certainly by default be high demand for such things, and profits to be made. OpenAI will ‘try to help’ in the sense that it is profitable to get involved. And by profitable, I somewhat mean profitable to OpenAI, but also I mean profitable to Sam Altman. This is an obvious way for him to cash in.
One must ask if this is in conflict with OpenAI’s non-profit mission, or when it would become so.
As usual, people say ‘competitiveness’ as if America was in non-zero danger of falling behind in such matters if we took our foot off the gas petal. This continues not to be the case. We are the dominant player. You can say good, let’s be even more dominant, and that is a valid argument, but do not pretend we are in danger.
Quiet Speculations
I noted last week that OpenAI’s study on GPT-4 and figuring out how to make biological weapons seemed to indeed indicate that it helped people figure out how to make such weapons, despite lacking statistical significance per se, and that the conclusion otherwise was misleading. Gary Marcus suggests that the reason they said it wasn’t significant in footnote C was that they did a Bonferroni correction that guards against fishing expeditions, except this was not a fishing expedition, so there should have been no correction. A variety of tests actually do show significance here, as does the eyeball test, and anti-p-hacking techniques were used to make this look otherwise, because this is the strange case where the authors were not positively inclined to find positive results. Gary is (as you would expect) more alarmed here than seems appropriate, but a non-zero amount of worry seems clearly justified.
Teortaxes suggests that data curation and pipelines are likely more important on the margin currently than architectural improvements, but no one pays them proper mind. Data is one of the places everyone is happy to keep quiet about, and proper curation and access could be a lot of the secret sauce keeping the big players ahead. If so, this could bode badly for compute limits, and it could explain why it seems relatively easy to do good distillation work and very difficult to match the big players.
Emmett Shear again says that if we create AGI, it needs to be a partner whose well-being we care about the way it cares about us. He is saying the RLHF-style approach won’t work, also presumably (based on what else he has said) that it would not be the right thing to do even if it did work. And if either of these are true, of course, then do not build that.
There are advantages to, if we can pull it off, making systems that are powerful enough to help us learn but not powerful enough to be a threat. Seems hard to hit that target. And yes, it is those who favor the humans who are the ones who should worry.
Nabeel Quereshi speaks of Moore’s Law for Intelligence, notes that we may not need any additional insights to reach the ‘inflection point’ of true self-improvement, although he does not use the word recursive. Says that because algorithms and data and compute will improve, any caps or pauses would be self-defeating, offers no alternatives that would allow humanity or value to survive. There is a missing mood.
Research scientist at DeepMind updates their timelines:
The update seems implausible in its details, pointing to multiple distinct cognitive calculations potentially going on. The new timeline is actually saying something pretty distinct about the curve of plausible outcomes, and it gets weirder the more I think about its details.
Vitalik on the Intersection AI and Crypto
Vitalik discusses potential interactions of AI and crypto, beyond the existing use case of arbitrage bots turning everything into an exploitative dark forest even more efficiently than they did before.
I would say it is even worse than this. If you accept that AI rulings happen in a ‘code is law’ style situation, even if we assume the AI fully remains a tool, we have to worry not only about adversarial attacks but also about all the backdoors and other strange behaviors, intentional and unintentional. Corner cases will inevitably get exploited. I really, really do not think going here is a good idea. LLMs make mistakes. Crypto is about, or needs to be about, systems that can never, ever make a mistake. Vitalik explores using ‘crypto magic’ to fix the issue but explains this will at best be expensive and hard, I think the problems are worse than he realizes.
In general, this all seems like classic crypto problems, where you are trying to solve for parts of the problem that are unlikely to be either necessary or sufficient for practical use cases. He asks, can we do better than the already-dystopian ‘centralized’ world? Here, ‘centralized’ seems to be a stand-in for ‘a human or alliance of humans can choose to determine the final outcome, and fix things if they go awry.’ And my answer to that is that removing that is unlikely to end well for the humans, even if the existential-level dangers are avoided.
Richard Ngo speculates that followers and follower counts will become important “currencies” in the future, as AI makes physical goods and intellectual labor abundant. Then you can cash this in for things you want, or for money. This will make it vitally important to crack down on fake followers and bot accounts.
This seems implausible to me, a kind of attachment to the present moment, as stated. Certainly, to the extent that humans remain in charge or even able to continue being humans, real human connection, ability to get attention where it matters, will matter. But what matters are the people you want. Why should you care about a bot army? What good is it to buy fake followers, will people actually get meaningfully fooled?
I would also say that the ability to fake such things meaningfully depends on people using naive counts rather than a robust analysis. Twitter lists exactly who is following who. There are already services that attempt to control for such issues, as I’m sure the platforms attempt to do themselves as well. AI will only supercharge what can be done there.
The Quest for Sane Regulations
France reluctantly agrees to support the AI Act, but makes clear it intends to weaken all the useful portions as much as it can during the implementation phase.
The Week in Audio
It was quite the week in audio, with two podcasts that I covered in extensive detail.
Dwarkesh Patel talked with Tyler Cowen, which I analyze here. This one was excellent. I recommend either listening or reading the analysis, ideally both. I disagree with Tyler’s views of transformative AI, and try to get into that more here, along with the places where I think his model is less different from mine than it appears. The parts about mundane AI and other things we are broadly in agreement but I have many thoughts.
Based Beff Jezos debated Connor Leahy, which I analyze here. Only listen to this one if this kind of debate is relevant to your interests, it is overall quite long and goes around in circles a lot, but it does contain actual arguments and claims that are important, and raises lots of good questions. Reading the summaries in my analysis is likely the way to go for most of you.
Tyler Cowen also sat down for this chat with Dan Shipper about using ChatGPT.
Some quick notes:
Rhetorical Innovation
Another round of Yudkowsky and Belrose disputing what was said in the past and what has and hasn’t been falsified, for those who care.
Originally in another context, but a very good principle:
An LLM can and will consider the entire internet, and all the data available. I noted this possibility right away with Sydney and Bing: If the primary way we search information begins responding in ways that depend on everything we say, then everything we say gets influenced by that consideration. And this could easily spiral way out of our control. Notice what SEO has already done to the internet.
Aligning a Dumber Than Human Intelligence is Still Difficult
How to train your own sleeper agent LLM, similar to the sleeper agent paper. Unfortunately this does not provide sufficient instructions for someone like me to be able to do this. Anyone want to help out? I have some ideas I’d like to try at some point.
A paper called ‘Summon a Demon and Bind It: A Grounded Theory of LLM Red Teaming in the Wild.’ This is about the people, more than about the tech, sounds like.
People Are Worried About AI, Many People
Guess who said this earlier this week, answering a question about another topic:
In case the details did not give it away, that was Donald Trump.
Wise words indeed. The public fears and opposes AI and the quest to build AGI. That is in part because there is a very clear, intuitive, instinctive, simple case anyone can understand, that perhaps building things smarter than us is not a good idea. That is also in large part because there is already scary stuff happening.
Donald Trump is focused, as always, on the issues near and dear to him. Someone trying to fake his endorsement, or potentially twisting his words, very much will get this man’s attention. And yes, he always will talk in this vague, vibe-driven, Simulacra-4 style, where there are no specific prescriptions on what to do, but ‘something has to be done fast.’ Here, it turns out to be exactly correct that no one knows what to do, that there might be no solution, although we have some ideas on where to start.
Does he understand the problems of existential risk? No, I presume he has no idea. Will he repeal Biden’s executive order without caring what is in it, merely because it is Biden’s? That seems likely.
Other People Are Not As Worried About AI Killing Everyone
Paul Graham asks good questions, although I worry about the answers.
Yes. AIs will not automatically have an instinct for self-preservation, although they will be learning to imitate any training data that includes instincts for self-preservation, so they will look like they have one sometimes and this will sometimes have that effect. However they will get such a self-preservation motive the moment they get a larger goal to accomplish (as in, ‘you can’t fetch the coffee if you’re dead’) and also there are various optimization pressures in favor of them getting a preference for self-preservation, as we have seen since Asimov. Things that have that preference tend to get preserved and copied more often.
I think we knew the answer to this back in 2016 in any case, because we had existence proofs. Some humans genuinely do not have a self-preservation instinct, and others actively commit suicide.
The Lighter Side
Only note is text bubbles still need some work. Love the meta. This is DALLE.
Obvious picture of the week, everyone who did not make it first is kicking themselves:
I will be getting a demo of the Apple Vision Pro today (February 8) at 11:30am at Grand Central, which is supposed to be 30 minutes long, followed by lunch at Strip House on 44th Street. If you would like, you can come join for any portion of that. I will doubtless report the results no matter what happens. Here is the prediction market on whether I buy one, price seems sane to me, early reports say productivity features are not there yet but entertainment is great, and I can see this going either way.
Questions you kind of wish that particular person wouldn’t ask?