Posts
Wikitag Contributions
I think it's a little more concerning that Dwarkesh has invested in this startup:
Mechanize is backed by investments from Nat Friedman and Daniel Gross, Patrick Collison, Dwarkesh Patel, Jeff Dean, Sholto Douglas, and Marcus Abramovitch.
And I do not see any disclosure of this in either the Youtube description or the Substack transcript at present.
3In that brief moment of uncertainty, anything could have happened. If one person had just packed up and left, everyone might have followed suit. But nobody reacted. Perhaps what kept the room still was the fear of being perceived as scared. Or the belief that surely, bad things could not happen to them. Or maybe they’d heard enough false alarms in their lives. I’m not sure.
One of the most depressing things about the Replication Crisis in especially social psychology is that many results from the 1950s and 1960s failed to replicate at all... except the Asch conformity experiments. Those seem to replicate just fine.
1At first glance, your linked document seems to match this. The herald who calls the printer "pig-headed" does so in direct connection with calling him "dull", which at least in modern terms would be considered a way of calling him stupid?
Not necessarily. 'Dull' can mean, in 1621 just as well as 2025, plenty of other things: eg "Causing depression or ennui; tedious, uninteresting, uneventful; the reverse of exhilarating or enlivening." (OED example closest in time: "Are my discourses dull? Barren my wit?" --Jonson's good friend & fellow playwright, William Shakespeare, Comedy of Errors (1623)); or, "Of persons, or their mood: Having the natural vivacity or cheerfulness blunted; having the spirits somewhat depressed; listless; in a state approaching gloom, melancholy, or sadness: the opposite of lively or cheerful." (Shakespeare again: "Sweet recreation barr'd, what doth ensue / But moodie and dull melancholly?") Which in the context of a 'dull' tradesman who refuses to hear the exciting news being brought by no less than 2 heralds before he knows 'the price', is sensible enough.
not reading your entire document?
That would certainly help, because if you read the rest of the Printer's rather cynical comments, constantly undermining the heralds, he doesn't sound in the slightest bit like he is supposed to be stupid or retarded - as opposed to a curmodgeonly critic constantly - obstinately, even - throwing water on a good time by sardonically remarking that he makes money by changing the dates on newspaper plates to print the old news as new news or mocking their talk of moonlight by noting that his telescope-maker has brought him moonshine before. (Not that printers, like Benjamin Franklin, were an occupation associated with low intelligence to begin with.)
OP's example is correct and you are wrong. 'Pigheaded' is neither a proposed root cause analysis nor does it mean 'are dumb'; perhaps you should check a dictionary before correcting others' usage. It means stubborn, strong-willed, obstinate, often to the point of foolishness or taking very harmful actions, or to quote the OED: "Having a head like that of a pig. Chiefly figurative: stupidly obstinate, perverse, or set in one's ways." Note: it is "stupidly obstinate", and not "stupid". This is because pigs are notoriously smart but stubborn: very strong, heavy, often hungry, whose mind can't easily be changed by an unfortunate swineherd or passerby in their way. (And this usage has been consistent since the start: the OED will give you the first attestation of it to Ben Jonson, where it describes a small-minded* printer who thinks that high-quality news has to be paid for, because that's how he operates; Jonson then mocks some other tradesmen for their own kinds of narrowmindedness, but not for any of them being low-IQ.) Hence, the Russell conjugation is correct: "pigheaded" is the highly insulting figurative term which intensifies the negative "obstinate" which is the bad version of the positive "firm". Just as 'firm' does not principally mean 'dumb', 'pigheaded' doesn't principally mean it either.
* note, by the way, that 'small-minded' doesn't mean, 'has a low cranial volume and thus lower than average intelligence', nor is it a root-cause analysis that their low intelligence is caused by inadequate neural tissue.
But the caveat there is that this is inherently a backwards-looking result:
We consider GPT-4o (OpenAI, 2024), Claude-3.5-Sonnet (Anthropic, 2024), Grok-2 (xAI, 2024), Gemini-1.5-Pro (Google, 2024), and DeepSeek-V3 (DeepSeek-AI, 2024).
So one way to put it would be that people & classifiers are good at detecting mid-2024-era chatbot prose. Unfortunately, somewhere after then, at least OpenAI and Google apparently began to target the problem of ChatGPTese (possibly for different reasons: Altman's push into consumer companion-bots/personalization/social-networking, and Google just mostly ignoring RLHF in favor of capabilities), and the chatbot style seems to have improved substantially. Even the current GPT-4o doesn't sound nearly as 4o-like as it did just back in November 2024. Since mode-collapse/ChatGPTese stuff was never a capabilities problem per se (just look at GPT-3!), but mostly just neglect/apathy on part of the foundation labs (as I've been pointing out since the beginning), it's not a surprise that it could improve rapidly once they put (possibly literally) any effort into fixing it.
Between the continued rapid increase in capabilities and paying some attention to esthetics & prose style and attackers slowly improving their infrastructure in the obvious ways, I expect over the course of 2025 that detecting prose from a SOTA model is going to get much more difficult. (And this excludes the cumulative effect on humans increasingly writing like ChatGPT.)
EDIT: today on HN, a post was on the front page for several hours with +70 upvotes, despite being blatantly new-4o-written (and impressively vapid). Is this the highest-upvoted LLM text on HN to date? I suspect that if it is, we'll soon see higher...
I'm not sure this is a big problem. How much net attrition do you really expect over a decade, say? By which point who really cares? You will have so much more AI progress, and accumulated data (particularly if you've been gradually replacing the lower-level employees and you have an 'automation wave' moving through the organization where employees increasingly train their automated replacements or their job is simply reorganizing the jobs to enable automation).
It seems like to the extent there's much attrition at high levels, it is reduced in considerable part by these very dynamics: as returns to high-level human labor go up, presumably, there is less attrition from voluntary retirement or leisure consumption (and if the returns go down, then that implies that there is no 'shortage' of people for such high-level positions and so no problem); and also as the remaining human work becomes more 'white-collar' and based on difficult-for-AI things like reputation or experience or ownership or creativity, aging or opportunity costs begin to matter less, reducing another source of attrition.
(Even if AI or robotics is unable to do the 'core' of a job, they can help deal with various obstacles which might prevent a human from doing the job. An elderly manager who might decide to retire in part because they are low-key becoming worried about safely driving to/from the office will no longer think about that when they have a self-driving car or remote working becomes ever more feasible; older managers who might be slipping in their grasp of details or who have 'senior moments' will be able to rely on AI secretaries to catch those or just pause stuff for a while until they're back to normal; elite women might invest more in careers if they have Claude-bot as a trustworthy nanny and chauffeur, etc. One is reminded of President Biden: his staffers were able to work around his issues by doing things like rescheduling or canceling events to avoid exposing him publicly when he was bad; it was only an event that even the POTUS can't arbitrarily schedule, a presidential debate, that punctured the carefully-constructed illusion. Few of those staffers were qualified to be President of the United States, and yet, you don't have to be a good president to observe "sounds like Joe's having a bad day today" and quietly cancel his evening appointments for him so he can get to bed early.)
Also notable: the big OpenAI reveal today was some sort of better personalization. Instead of the crude 'saved facts' personalization ChatGPT has had for a long time and which has never made much of a difference, they're doing... something. Unclear if it's merely RAG or if they are also doing something interesting like lightweight finetuning. But the GPTs definitely seem to have much better access to your other sessions in the web interface, and as far as I know, few other interfaces with frontier models have tried to do much personalization, so this will be an interesting real-world test at scale about how much simple personalization can help with LLMs (similar to Midjourney's relatively new personalization feature, which I get a lot out of).
I don't think this is true at all. How do you translate, say, rotating multiple shapes in parallel into text?
At least for multimodal LLMs in the pure-token approach like Gato or DALL-E 1 (and probably GPT-4o and Gemini, although few details have been published), you would be able to do that by generating the tokens which embody an encoded image (or video!) of several shapes, well, rotating in parallel. Then you just look at them.
Pursuit of novelty is not vnm-incoherent. Furthermore, it is an instrumentally convergent drive; power-seeking agents will seek novelty as well, because learning increases power in expectation (see: value of information).
Or to put it another way: any argument which convincingly proves that 'incoherent search processes ultimately outcompete coherent search processes' is also an argument which convinces a VNM agent to harness the superior incoherent search processes instead of the inferior coherent ones.
As defined, this is a little paradoxical: how could I convince a human like you to perceive domains of real improvement which humans do not perceive...?
See, this is exactly the example I would have given: truesight is an obvious example of a domain of real improvement which appears on no benchmarks I am aware of, but which appears to correlate strongly with the pretraining loss, is not applied anywhere (I hope), is unobvious that LLMs might do it and the capability does not naturally reveal itself in any standard use-cases (which is why people are shocked when it surfaces), and it would have been easy for no one to have observed it up until now or dismissed it, and even now after a lot of publicizing (including by yours truly), only a few weirdos know much about it.
Why can't there be plenty of other things like inner-monologue or truesight? ("Wait, you could do X? Why didn't you tell us?" "You never asked.")
Maybe a better example would be to point out that 'emergent' tasks in general, particularly multi-step tasks, can have observed success rates of precisely 0 in feasible finite samples, but extreme brute-force sampling reveals hidden scaling. Humans would perceive zero improvement as the models scaled (0/100 = 0%, 0/100 = 0%, 0/100 = 0%...), even though they might be rapidly improving from 1/100,000 to 1/10,000 to 1/1,000 to... etc. "Sampling can show the presence of knowledge but not the absence."