Adam Scherlis

GPT-175bee

Epistemic status: whimsical Bees: a new unit of measurement for ML model size Talking about modern ML models inevitably leads to a bunch of hard-to-intuit large numbers, especially when it comes to parameter count. To address this, we propose that we adopt a new, human-friendly unit to measure the number of learnable parameters in an architecture: 1 beepower = 1 BP = 1 billion parameters Bees have about one billion[1] synapses[2] in their forebrain[3], so this gives a nice basis for comparisons[4] between animal brains and artificial neural nets. Like horsepower and candlepower,[5] the unit of beepower expresses the scale of a new and unfamiliar technology in terms that we are already familiar with. And it makes discussion of model specs flow better. "This model has twenty bees", you might say. Or "wow, look at all that beepower; did you train long enough to make good use of it?" Here's a helpful infographic to calibrate you on this new unit of measurement: The parameter count of various recent language models, denoted in beepower. Other animals We can even benchmark[6] against more or less brainy animals. The smallest OpenAI API model, Ada, is probably[7] 350 million parameters, or about a third of a bee, which is comparable to a cricket: While Jiminy Cricket can compose better English than Ada, this cricket cannot. The next size up, Babbage, is around 1.3 BP, or cockroach-sized. Curie has almost seven bees, which is... sort of in an awkward gap between insects and mammals. Davinci is a 175-bee model, which gets us up to hedgehog (or quail) scale: As a large language model trained by OpenAI, I don't have the ability to "be the cutest little guy oh my gosh" Gopher (280 BP) is partridge (or ferret) sized. More research into actual gophers is needed to know how many gophers worth of parameters Gopher has. Really, they should've named Gopher "Partridge" or "Ferret"! Amusingly, PaLM, at 540 bees, has about as many parameters as a chinchilla has synapse

124Feb 8, 2023

Adam Scherlis

Message

1152

124

104

Utilitarian inequality metrics

TL;DR: Use Atkinson with ϵ=1 or generalized entropy with α=0. Inequality indices are numerical metrics of income (or wealth, etc) inequality. There are some downsides to society coordinating around a few canonical numerical metrics, like Goodhart's law; there are also big upsides, like making it harder to cherry-pick statistics, and...

Nov 13, 202525

Ternary plots are underrated

My post on the grapefruit-juice effect contains a ternary plot of citrus fruits. Here it is again (source): Ternary plots are great! They're used in a number of specialized fields, but I think they would be more widely popular if people were more familiar and comfortable with them. Let's make...

Nov 11, 202516

The grapefruit juice effect

The medication I'm taking for insomnia interacts badly with grapefruit juice. This isn't much of a issue, yet; the cravings are still manageable. I only dream about grapefruit sometimes. I was never the kind of person to blow my whole budget on the stuff. The biggest problem, really, is that...

Nov 10, 202536

n-ary Huffman coding

Huffman coding is a method for constructing optimal prefix codes! Codes and trees As previously alluded to on this blog, a code represents an implicit set of beliefs about the frequency distribution of different kinds of text. Longer codewords represent lower implied frequencies. Prefix codes give each symbol in the...

Nov 9, 202518

Review: Parsifal at the SF Opera

I saw Wagner's Parsifal a couple weeks ago with a bunch of Twitter opera nerds. Parsifal has a reputation among opera fans for being long, melodramatic, self-indulgent, and having a slow-moving plot; but given that these are opera fans, perhaps these should be taken as compliments. One should also note...

Nov 8, 202510

GPTF-8: A tokenizer-based character encoding

There are two steps to any byte-based character encoding. The first and much more interesting step is the translation from written language -- in all its chaotic glory -- to a fixed inventory of "characters", from which any string can be built up as a sequence. In the modern day,...

Nov 7, 20257

Review: K-Pop Demon Hunters (2025)

(This review contains spoilers for the entire plot of the film.) K-Pop Demon Hunters is a very popular movie. It is the first Netflix movie to hit #1 at the box office. It is "the first film soundtrack on the Billboard Hot 100 to have four of its songs in...

Nov 6, 202520

Load More (7/30)

LESSWRONG
LW

LESSWRONG
LW

Adam Scherlis

Adam Scherlis

Adam Scherlis

GPT-175bee

Inner Misalignment in "Simulator" LLMs

A hundredth of a bit of extra entropy

New Frontiers in Mojibake

Adam Scherlis

Utilitarian inequality metrics

Ternary plots are underrated

The grapefruit juice effect

n-ary Huffman coding

Review: Parsifal at the SF Opera

GPTF-8: A tokenizer-based character encoding

Review: K-Pop Demon Hunters (2025)

Utilitarian inequality metrics

Ternary plots are underrated

The grapefruit juice effect

n-ary Huffman coding

Review: Parsifal at the SF Opera

GPTF-8: A tokenizer-based character encoding

Review: K-Pop Demon Hunters (2025)

GPT-175bee

Inner Misalignment in "Simulator" LLMs

A hundredth of a bit of extra entropy

New Frontiers in Mojibake