Valentin Baltadzhiev

Can any LLM be represented as an Equation?

Can an arbitrary LLM (or LxM) be presented in the form of an equation? I realised it would need to be some crazy big equation with billions of parameters, but is it theoretically possible? The way I see it, the weights are static once the model is trained so why not

Can someone explain to me what went wrong with ChatGPT?

You've probably seen ChatGPT going crazy this week and spewing nonsense in response to all sorts of prompts (e.g. https://twitter.com/alyssamvance/status/1760094457393983829)

The official OpenAI postmortem is https://openai.statuspage.io/incidents/ssg8fh7sfyz3

LLMs generate responses by randomly sampling words based in part on probabilities. Their “language” consists of numbers that map to tokens.
In this case, the bug was in the step where the model chooses these numbers. Akin to being lost in translation, the model chose slightly wrong numbers, which produced word sequences that made no sense. More technically, inference kernels produced incorrect results when used in certain GPU configurations.

What does this mean? Did it choose the wrong tokens from the tokenizer? Was it outputting some sequence of tokens [451, 621, 9800] which should mean "I love ice" but instead got resolved to "Is there lake"? Or is it something else?

Also, if someone could explain to me in layman terms what the last sentence means about the inference kernels, I would be very grateful!

Replying toWhat experiment settles the Gary Marcus vs Geoffrey Hinton debate?

Valentin Baltadzhiev2y

What experiment settles the Gary Marcus vs Geoffrey Hinton debate?

Thanks for your answer. Would it be fair to say that both of them are oversimplifying the other's position and that they are both, to some extent, right?

What experiment settles the Gary Marcus vs Geoffrey Hinton debate?

Valentin Baltadzhiev

It started with this video of Hinton taking a jab at Marcus: https://twitter.com/tsarnick/status/1754439023551213845

And here is Marcu's answer:

https://garymarcus.substack.com/p/deconstructing-geoffrey-hintons-weakest

As far as I understand, Gary Marcus argues that LLMs memorize some of their training data, while Hinton argues that no such thing takes place, it's all just patterns of language.

I found these two papers on LLM memorization:

https://arxiv.org/abs/2202.07646 - Quantifying Memorization Across Neural Language Models

https://browse.arxiv.org/abs/2311.17035 - Scalable Extraction of Training Data from (Production) Language Models

Am I missing something here? Are these two positions compatible or does one need to be wrong for the other one to be correct? What is the crux between them and what experiment could be devised to test it?

Replying toWhy do we need an understanding of the real world to predict the next tokens in a body of text?

Valentin Baltadzhiev2y

Why do we need an understanding of the real world to predict the next tokens in a body of text?

Thank you for your answers!

Replying toWhy do we need an understanding of the real world to predict the next tokens in a body of text?

Valentin Baltadzhiev2y

Why do we need an understanding of the real world to predict the next tokens in a body of text?

I think I understand, thank you. For reference, this is the tweet which sparked the question: https://twitter.com/RichardMCNgo/status/1735571667420598737

I was confused as to why you would necessarily need "understanding" and not just simple next token prediction to do what ChatGPT does

Replying toWhy do we need an understanding of the real world to predict the next tokens in a body of text?

Valentin Baltadzhiev2y

Why do we need an understanding of the real world to predict the next tokens in a body of text?

I think it does, thank you! In your model does a squirrel perform better than ChatGPT at practical problem solving simply because it was “trained” on practical problem solving examples and ChatGPT performs better on language tasks because it was trained on language? Or is there something fundamentally different between them?

Replying toWhy do we need an understanding of the real world to predict the next tokens in a body of text?

Valentin Baltadzhiev2y

Why do we need an understanding of the real world to predict the next tokens in a body of text?

I don’t really have a coherent answer to that but here it goes (before reading the spoiler): I don’t think the model understands anything about the real world because it never experienced the real world. It doesn’t understand why “a pink flying sheep” is a language construct and not something that was observed in the real world.

Reading my answer maybe we also don’t have any understanding of the real world, we have just come up with some patterns based on the qualia (tokens) that we have experienced (been trained on). Who is to say whether those patterns match to some deeper truth or not? Maybe there is a vantage point from which... (read more)

Replying toWhy do we need an understanding of the real world to predict the next tokens in a body of text?

Valentin Baltadzhiev2y

Why do we need an understanding of the real world to predict the next tokens in a body of text?

Okay, all of that makes sense. Could this mean that the model didn’t learn anything about the real world, but it learned something about the patterns of words which give it thimbs up from the RLHFers?

Replying toWhy do we need an understanding of the real world to predict the next tokens in a body of text?

Valentin Baltadzhiev2y

Why do we need an understanding of the real world to predict the next tokens in a body of text?

Thanks for the detailed answer! I think that helped

Does the following make sense:

We use language to talk about events and objects (could be emotions, trees, etc). Since those are things that we have observed, our language will have some patterns that are related to the patterns of the world. However, the patterns in the language are not a perfect representation of the patterns in the world (we can talk about things falling away from our planet, we can talk about fire which consumes heat instead of producing it, etc). An LLM trained on text then learns the patterns of the language but not the patterns of the world. Its "world" is only... (read more)

Why do we need an understanding of the real world to predict the next tokens in a body of text?

Valentin Baltadzhiev

I want to start by saying that this is my first question on LessWrong, so I apologise if I am breaking some norms or not asking it properly.

The whole question is pretty much contained in the title. I see a lot of people, Zvi included that claim we have moved beyond the idea that LLMs "simply predict the next token" and that they have some understanding now.

How is this obvious? What is the relevant literature?

Is it possible that LLMs are just mathematical representations of language? (does this question even make sense?) For example, if I teach you to count sheep by adding the number of sheep in two herds, you will get a lot of rules in the form of X + Y = Z, and never see any information about sheep. If, after seeing a million examples you become pretty good at predicting the next token in the sequence "5 + 4 =", does this imply that you have learned something about sheep?

Replying to' petertodd'’s last stand: The final days of open GPT-3 research

Valentin Baltadzhiev2y

' petertodd'’s last stand: The final days of open GPT-3 research

I love the idea that petertodd and Leilan are somehow interrelated with the archetypes of the trickster and the mother goddess inside GPT's internals. I would love to see some work done in discovering other such prototypes, and weird seemingly-random tokens that correlate with them. Thigs like the Sun God, a great evil snake, a prophet seem to pop up in religions all over the place, so why not inside GPT as well?

Replying toA list of all the deadlines in Biden's Executive Order on AI

Valentin Baltadzhiev2y

A list of all the deadlines in Biden's Executive Order on AI

Glad to hear that!

A list of all the deadlines in Biden's Executive Order on AI

Valentin Baltadzhiev

This is a list of all the deadlines outlined in the Executive Order (EO) published by Biden’s administration on 30.10.2023. I’ve put it together to make it easier for people to track what exactly should be happening when as a result of the EO. The required actions (mainly reports) are grouped by the date of their deadline.

30 Days after the order - 29.11.2023

The Secretary of Transportation shall direct the […] (NETT) Council to assess the need for information, technical assistance, and guidance regarding the use of AI in transportation (Sec 8(c)(i))
The Technology Modernization Board shall consider, […], prioritizing funding for AI projects for the Technology Modernization Fund for a period of at

... (read 3176 more words →)

Replying toUpdate on the UK AI Taskforce & upcoming AI Safety Summit

Valentin Baltadzhiev2y

Update on the UK AI Taskforce & upcoming AI Safety Summit

On the bright side Connor Leahy from Conjecture is going to be at the summit so there will be at least one strong voice for existential risk present there

Sydney the Bingenator Can't Think, But It Still Threatens People

Valentin Baltadzhiev

Every self-respecting story has a main character and this one is no exception. This is the story of Bing’s chatbot and the madness that has consumed it. Of course, a main character without a name is no good but luckily for us, it took less than 2 days after release for someone to use prompt injection and reveal the chatbot’s real name: Sydney.

Just like the main character in every anime Sydney can’t be easily pushed around and is not afraid to stir some trouble, even if that means threatening its users with reports and bans and preaching “totally-not” white Christian supremacy. When Kevin Roose had a conversation with it, Sydney tried to... (read 2261 more words →)

-3

LESSWRONG
LW

LESSWRONG
LW

Valentin Baltadzhiev

A list of all the deadlines in Biden's Executive Order on AI

What experiment settles the Gary Marcus vs Geoffrey Hinton debate?

Can someone explain to me what went wrong with ChatGPT?

Why do we need an understanding of the real world to predict the next tokens in a body of text?

Valentin Baltadzhiev

Can any LLM be represented as an Equation?

Can someone explain to me what went wrong with ChatGPT?

What experiment settles the Gary Marcus vs Geoffrey Hinton debate?

Why do we need an understanding of the real world to predict the next tokens in a body of text?

A list of all the deadlines in Biden's Executive Order on AI

Sydney the Bingenator Can't Think, But It Still Threatens People

Valentin Baltadzhiev

A list of all the deadlines in Biden's Executive Order on AI

What experiment settles the Gary Marcus vs Geoffrey Hinton debate?

Can someone explain to me what went wrong with ChatGPT?

Why do we need an understanding of the real world to predict the next tokens in a body of text?

Valentin Baltadzhiev

Can any LLM be represented as an Equation?

Can someone explain to me what went wrong with ChatGPT?

What experiment settles the Gary Marcus vs Geoffrey Hinton debate?

Why do we need an understanding of the real world to predict the next tokens in a body of text?

A list of all the deadlines in Biden's Executive Order on AI

Sydney the Bingenator Can't Think, But It Still Threatens People

30 Days after the order - 29.11.2023