Posts
Wiki Contributions
DeepSeek is Chinese. I'm not really familiar with the company.
DeepSeek is the best Chinese DL research group now and have been for at least a year. If you are interested in the topic, you ought to learn more about them.
I thought Chinese companies were at least a year behind the frontier.
This seems roughly consistent with what you would expect. People usually say half a year to a year behind. Q* was invented somewhere in summer 2023, according to the OA coup reporting; ballpark June-July 2023, I got the impression since it seemed to already be a topic of discussion with the Board c. August 2023 pre-coup. Thus, we are now (~20 Nov 2024) almost at December 2024, which is about a year and a half. o1-preview was announced 12 September 2024, 74 days ago, and o1-preview's benchmarks were much worse than the true o1 which was still training then (and of course, OA has kept improving it ever since, even if we don't know how - remember, time is always passing†, and what you read in a blog post may already be ancient history). Opensource/competitor models (not just Chinese or DeepSeek specifically) have a long history of disappointing in practice when they turn out to be much narrower, overfit to benchmarks, or otherwise somehow lacking in quality & polish compared to the GPT-4s or Claude-3s.
So, if a competing model claims to match o1-preview from almost 3 months ago, which itself is far behind o1, with additional penalties from compensating for the hype and the apples-to-oranges comparisons, and where we still don't know if they are actually the same algorithm at core (inasmuch as neither OA nor DeepSeek, AFAIK, have yet to publish any kind of detailed description of what Q*/Strawberry/r1 is), and possibly worst-case as much as >1.5 years behind if DS has gone down a dead end & has to restart...
Overall, this still seems roughly what you would expect now: 'half a year to a year behind'. It's always a lot easier to catch up with an idea after someone else has proven it works and given you an awful lot of hints about how it probably works, like the raw sample transcripts. (I particularly note the linguistic tics in this DS version too, which I take as evidence for my inner-monologue splicing guess of how the Q* algorithm works.)
† I feel very silly pointing this out: that time keeps passing, and if you think that some new result is startling evidence against the stylized fact "Chinese DL is 6-12 months behind" that you should probably start by, well, comparing the new result to the best Western DL result 6–12 months ago! This should be too obvious to even mention. / And yet, I constantly get the feeling that people have been losing their sort of... "temporal numeracy", for lack of a better phrase. That they live in a 'Big Now' where everything has happened squashed together. In the same way that in politics/economics, people will talk about the 1980s or 1990s as if all of that was just a decade ago instead of almost half a century ago (yes, really: 2024 − 1980 = 44), many people discussing AI seems to have strangely skewed mental timelines. / They talk like GPT-4 came out, like, a few months after GPT-3 did, maybe? So GPT-5 is wildly overdue! That if a Chinese video model matches OA Sora tomorrow, well, Sora was announced like, a month ago, something like that? So they've practically caught up! OA only just announced o1, and DeepSeek has already matched it! Or like 2027 is almost already here and they're buying plane tickets for after Christmas. There's been a few months without big news? The DL scaling story is over for good and it hit the wall!
2I had a related one as a teenager: there are various expressions about women being too beautiful to look at or that it hurt to look at, etc. I thought they were all overwrought literary expressions - a woman you loved or had a crush on, sure, that's love, but just a random woman? - until I went to dinner in a group which happened to include such a woman.
(Thankfully, being a large group in a noisy restaurant, I could get away with not looking at her all evening; although I got a little angry this could even be a thing - I never signed up for that! I've wondered if or how long it'd take for that to wear off, but I never saw her again, so I have no idea.)
I don't believe there are any details about the restructuring, so a detailed analysis is impossible. There have been a few posts by lawyers and quotes from lawyers, and it is about what you would expect: this is extremely unusual, the OA nonprofit has a clear legal responsibility to sell the for-profit for the maximum $$$ it can get or else some even more valuable other thing which assists its founding mission, it's hard to see how the economics is going to work here, and aspects of this like Altman getting equity (potentially worth billions) render any conversion extremely suspect as it's hard to see how Altman's handpicked board could ever meaningfully authorize or conduct an arms-length transaction, and so it's hard to see how this could go through without leaving a bad odor (even if it does ultimately go through because the CA AG doesn't want to try to challenge it).
I did not understand his response at all, and it sounds like I would have to reread a bunch of Turntrout posts before any further comment would just be talking past each other, so I don't have anything useful to say. Maybe someone else can re-explain his point better and why I am apparently wrong.
If you simply search the title, you will find many PDFs: https://scholar.google.com/scholar?q=Rule Breaking%2C Honesty%2C and Migration (eg)
For perhaps obvious reasons, I thought "The GPT" was by far the best one.
Also worth noting is Steve Hsu's recent discussion of his meetings with China VC, government, researchers etc. reporting from on the ground in Shanghai and Beijing etc: https://www.manifold1.com/episodes/letter-from-shanghai-reflections-on-china-in-2024-73/transcript
Hsu is a long-time China hawk and has been talking up the scientific & technological capabilities of the CCP for a long time, saying they were going to surpass the West any moment now, so I found this interesting when Hsu explains that:
-
the scientific culture of China is 'mafia' like (Hsu's term, not mine) and focused on legible easily-cited incremental research, and is against making any daring research leaps or controversial breakthroughs...
but is capable of extremely high quality world-class followup and large scientific investments given a clear objective target and government marching orders
-
there is no interest or investment in an AI arms race, in part because of a "quiet confidence" (ie. apathy/laying-flat) that if anything important happens, China can just catch up a few years later and win the real race. They just aren't doing it. There is no Chinese Manhattan Project. There is no race. They aren't dumping the money into it, and other things, like chips and Taiwan and demographics, are the big concerns which have the focus from the top of the government, and no one is interested in sticking their necks out for wacky things like 'spending a billion dollars on a single training run' without explicit enthusiastic endorsement from the very top.
Let the crazy Americans with their fantasies of AGI in a few years race ahead and knock themselves out, and China will stroll along, and scoop up the results, and scale it all out cost-effectively and outcompete any Western AGI-related stuff (ie. be the BYD to the Tesla). The Westerners may make the history books, but the Chinese will make the huge bucks.
So, this raises an important question for the arms race people: if you believe it's OK to race, because even if your race winds up creating the very race you claimed you were trying to avoid, you are still going to beat China to AGI (which is highly plausible, inasmuch as it is easy to win a race when only one side is racing), and you have AGI a year (or two at the most) before China and you supposedly "win"... Then what?
- race to AGI and win
- trigger a bunch of other countries racing to their own AGI (now that they know it's doable, increasingly much about how to do it, can borrow/steal/imitate the first AGI, and have to do so "before it's too late")
- ???
- profit!
What does winning look like? What do you do next? How do you "bury the body"? You get AGI and you show it off publicly, Xi blows his stack as he realizes how badly he screwed up strategically and declares a national emergency and the CCP starts racing towards its own AGI in a year, and... then what? What do you do in this 1 year period, while you still enjoy AGI supremacy? You have millions of AGIs which can do... stuff. What is this stuff? Are you going to start massive weaponized hacking to subvert CCP AI programs as much as possible short of nuclear war? Lobby the UN to ban rival AGIs and approve US carrier group air strikes on the Chinese mainland? License it to the CCP to buy them off? Just... do nothing and enjoy 10%+ GDP growth for one year before the rival CCP AGIs all start getting deployed? Do you have any idea at all? If you don't, what is the point of 'winning the race'?
(This is a question the leaders of the Manhattan Project should have been asking themselves when it became obvious that there were no genuine rival projects in Japan or Germany, and the original "we have to beat Hitler to the bomb" rationale had become totally irrelevant and indeed, an outright propaganda lie. The US got The Bomb, immediately ensuring that everyone else would be interested in getting the bomb, particularly the USSR, in the foreseeable future... and then what? Then what? "I'll ask the AGIs for an idea how to get us out of this mess" is an unserious response, and it is not a plan if all of the remaining viable plans the AGIs could implement are one of those previous plans which you are unwilling to execute - similar to how 'nuke Moscow before noon today' was a viable plan to maintain nuclear supremacy, but wasn't going to happen, and it would have been better to not put yourself in that position in the first place.)
1
1I'm not sure I see any difference here between regular dangerously convergent instrumental drives and this added risk of 'intrinsic' drives. They just seem like the same thing to me. Like the two predictions you give seem already true and fulfilled:
Relative to other goals, agentic systems are easy to steer to seek power.
Agentic systems seek power outside of the “training distribution”, but in ways which don’t seem to be part of larger power-seeking plans.
Both of these seem like I would expect from a flexible, intelligent agent which is capable of handling many complicated changing domains, like a LLM: they are easy to steer to seek power (see: all the work on RLHF and the superficiality of alignment and ease of steering and low-dimensional embeddings), and they can execute useful heuristics even if those cannot be easily explained as part of a larger plan. (Arguably, that's most of what they do currently.) In the hypotheticals you give, the actions seem just like a convergent instrumental drive of the sort that an agent will rationally develop in order to handle all the possible tasks which might be thrown at it in a bewildering variety of scenarios by billions of crazy humans and also other AIs. Trying to have 'savings' or 'buying a bit of compute to be safe', even if the agent cannot say exactly what it would use those for in the current scenario, seems like convergent, and desirable, behavior. Like buying insurance or adding validation checks to some new code, usually it won't help, but sometimes the prudence will pay off. As humans say, "shit happens". Agents which won't do that and just helplessly succumb to hardware they know is flaky or give up the moment something is a little more than expensive than average or write code that explodes the instant you look at it funny because you didn't say "make sure to check for X Y & Z" - those agents are not good agents for any purpose.
If there are 'subshards' which achieve this desirable behavior because they, from their own perspective, 'intrinsically' desire power (whatever that sort of distinction makes when you've broken things down that far), and it is these subshards which implement the instrumental drive... so what? After all, there has to be some level of analysis at which an agent stops thinking about whether or not it should do some thing and just starts doing the thing. Your muscles "intrinsically desire" to fire when told to fire, but the motor actions are still ultimately instrumental, to accomplish something other than individual muscles twitching. You can't have 'instrumental desire' homunculuses all the way down to the individual transistor or ReLU neuron.
Possibly it will still be counterintuitive to many folks, as Said quoted in a sibling comment.
No, this is a little different. Your approach here sounds like ours and the intuitive one (just at the cost of additional complexity).
The 'auto dark mode' we abandoned is where you just use step #2 there and you skip #1 (and thus, any widget or toggle which enables a reader to do anything with localStorage), and 'auto is the only state'. The logic there is, the reader already has access to a widget or toggle to set their dark mode preference: it's just their OS/browser, which will have some config page somewhere with various settings like 'turn on dark mode at night' or 'always use dark mode' or 'always use light mode'. Just trust the OS/browser and use whatever setting it sends to the web page. Don't waste the effort and screen real estate to add in a redundant widget/toggle. It's handled already. Easier for everyone - it Just Works™!
Unfortunately, the connection between 'a year ago when I updated my Android phone and it asked me if I wanted to use the cool new dark mode, I said yes' and 'this webpage I am reading now is in dark mode for some reason, and I can't change it back to normal???', apparently winds up eluding some readers. (This is what Said's sibling comment is about.) It winds up being "too much magic".
The current toggle+localStorage+auto approach, on the other hand, while adding to the clutter, does not seem to confuse readers: "the page is in dark-mode, for some reason. But I want want light-mode and I am unhappy. I see a little light-mode button. I push that button. Now the page is in light-mode. I am happy." (And then it is light-mode ever after.) At least, I have seen many fewer (or no) complaints about the dark mode being on when it shouldn't be after we moved to the toggle. So as far as we can tell, it's working.
(Not "idle worship"?)