LESSWRONG
LW

All of Valentin Baltadzhiev's Comments + Replies

What experiment settles the Gary Marcus vs Geoffrey Hinton debate?
Valentin Baltadzhiev10

Thanks for your answer. Would it be fair to say that both of them are oversimplifying the other's position and that they are both, to some extent, right?

Reply
2[anonymous]
Yes. Also it says on Marcus's Wikipedia page he founded 2 AI startups, one in 2015 and another in 2019. It is an unreasonable view to believe he doesn't understand transformers. Marcus tried other techniques that evidently didn't work well enough (SOTA for robotics is now a variation on transformers). Hinton seems to have partial credit for back propagation and recent credit for capsule networks which work but also didn't work well enough. Of course Marcus also calls Hinton old in his response, Hinton is 24 years older.
Why do we need an understanding of the real world to predict the next tokens in a body of text?
Valentin Baltadzhiev10

I think I understand, thank you. For reference, this is the tweet which sparked the question: https://twitter.com/RichardMCNgo/status/1735571667420598737

I was confused as to why you would necessarily need "understanding" and not just simple next token prediction to do what ChatGPT does

Reply
Why do we need an understanding of the real world to predict the next tokens in a body of text?
Valentin Baltadzhiev10

I think it does, thank you! In your model does a squirrel perform better than ChatGPT at practical problem solving simply because it was “trained” on practical problem solving examples and ChatGPT performs better on language tasks because it was trained on language? Or is there something fundamentally different between them?

Reply
1Random Developer
I suspect ChatGPT 4's weaknesses come from several sources, including: 1. It's effectively amnesiac, in human terms. 2. If you look at the depths of the neural networks and the speed with which they respond, they have more in common with human reflexes than deliberate thought. It's basically an actor doing a real-time improvisation exercise, not a writer mulling over each word. The fact that it's as good as it is, well, it's honestly terrifying to me on some level. 3. It has never actually lived in the physical world, or had to solve practical problems. Everything it knows comes from text or images. Most people's first reaction to ChatGPT is to overestimate it. Then they encounter various problems, and they switch to underestimating it. This is because we're used to interacting with humans. But ChatGPT is very unlike a human brain. I think it's actually better than us at some things, but much worse at other key things.
Why do we need an understanding of the real world to predict the next tokens in a body of text?
Valentin Baltadzhiev10

I don’t really have a coherent answer to that but here it goes (before reading the spoiler): I don’t think the model understands anything about the real world because it never experienced the real world. It doesn’t understand why “a pink flying sheep” is a language construct and not something that was observed in the real world.

Reading my answer maybe we also don’t have any understanding of the real world, we have just come up with some patterns based on the qualia (tokens) that we have experienced (been trained on). Who is to say whether those patterns m... (read more)

Reply
9Random Developer
You're asking good questions! Let me see if I can help explain what other people are thinking. When talking about cutting edge models, you might want to be careful when making up examples like this. It's very easy to say "LLMs can't do X", when in fact a state-of-the-art model like GPT 4 can actually do it quite well. For example, here's what happens if you ask ChatGPT about "pink flying sheep". It realizes that sheet are not supposed to be pink or to fly. So it proposes several hypotheses, including: 1. Something really weird is happening. But this is unlikely, given what we know about sheep. 2. The observer might be on drugs. 3. The observer might be talking about a work of art. 4. Maybe someone has disguised a drone as a pink flying sheep. ...and so on. Now, ChatGPT 4 does not have any persistent memories, and it's pretty bad at planning. But for this kind of simple reasoning about how the world works, it's surprisingly reliable. For an even more interesting demonstration of what ChatGPT can do, I was recently designing a really weird programming language. It didn't work like any popular language. It was based on a mathematical notation for tensors with implicit summations, it had a Rust-like surface syntax, and it ran on the GPU, not the CPU. This particular combination of features is weird enough that ChatGPT can't just parrot back what it learned from the web. But when I have ChatGPT a half-dozen example programs in this hypothetical language, it was perfectly capable of writing brand-new programs. It could even recognize the kind of problems that this language might be good for solving, and then make a list of useful programs that couldn't be expressed in my language. It then implemented common algorithms in the new language, in a more or less correct fashion. (It's hard to judge "correct" in a language that doesn't actually exist.) I have had faculty advisors on research projects who never engaged at this level. This was probably because they cou
Why do we need an understanding of the real world to predict the next tokens in a body of text?
Valentin Baltadzhiev10

Okay, all of that makes sense. Could this mean that the model didn’t learn anything about the real world, but it learned something about the patterns of words which give it thimbs up from the RLHFers?

Reply
3[anonymous]
This becomes a philosophical question. I think you are still kinda stuck on a somewhat obsolete view of intelligence. For decades people thought that AGI would require artificial self awareness and complex forms of metacognition. This is not the case. We care about correct answers on tasks, where we care the most about "0 shot" performance on complex real world tasks. This is where the model gets no specific practice on the task, just a lot of practice to develop the skills needed for each step needed to complete the task, and access to reference materials and a task description. For example if you could simulate an auto shop environment you might train the model to rebuild Chevy engines and have a 0 shot test on rebuilding Ford engines. Any model that scores well on a benchmark of Ford rebuilds has learned a general understanding about the real world. Then imagine you don't have just 1 benchmark, but 10,000+, over a wide range of real world 0 shot tasks. Any model that can do at human level, across as many of these tests as the average human (derived with sampling since a human taking the test suite doesn't have time for 10,000 tests) is an AGI. Once the model beats all living humans at all tasks in expected value on the suite the model is ASI. It doesn't matter how it works internally, the scores on a realistic unseen test is proof the model is AGI.
Why do we need an understanding of the real world to predict the next tokens in a body of text?
Valentin Baltadzhiev10

Thanks for the detailed answer! I think that helped

Does the following make sense:

We use language to talk about events and objects (could be emotions, trees, etc). Since those are things that we have observed, our language will have some patterns that are related to the patterns of the world. However, the patterns in the language are not a perfect representation of the patterns in the world (we can talk about things falling away from our planet, we can talk about fire which consumes heat instead of producing it, etc). An LLM trained on text then learns the ... (read more)

Reply
2[anonymous]
I think what you are saying simply generalizes to: "The model learns only what is has to learn to minimize loss in the training environment". Every weight is there to minimize loss. It won't learn a deep understanding of anything when a shallow understanding works. And this is somewhat true. It's more complex than this and depends heavily on the specific method used - architecture, hyperparams, distillation or sparsity. In practice, what does this mean? How do you get the model to have a deep understanding of a topic? In the real world you mostly get this by being forced to have it, aka actually doing a task in a domain. Note that this is the second step to LLM training, rounds of RLHF that teach the model to do the task of question answering, or RL against another LLM to filter out unwanted responses. This should also work for real world task domains.
' petertodd'’s last stand: The final days of open GPT-3 research
Valentin Baltadzhiev21

I love the idea that petertodd and Leilan are somehow interrelated with the archetypes of the trickster and the mother goddess inside GPT's internals. I would love to see some work done in discovering other such prototypes, and weird seemingly-random tokens that correlate with them. Thigs like the Sun God, a great evil snake, a prophet seem to pop up in religions all over the place, so why not inside GPT as well?

Reply
Update on the UK AI Taskforce & upcoming AI Safety Summit
Valentin Baltadzhiev63

On the bright side Connor Leahy from Conjecture is going to be at the summit so there will be at least one strong voice for existential risk present there

Reply
Bing Chat is blatantly, aggressively misaligned
Valentin Baltadzhiev10

For what it’s worth it’s probably a good thing that the Bing chatbot is like that. The overall attitude towards AI for the last few months has been one of unbridled optimism and people seeing a horribly aligned model in action might be a wake up call for some, showing that the people deploying those models are unable to control them.

Reply
Sydney the Bingenator Can't Think, But It Still Threatens People
Valentin Baltadzhiev10

You make interesting points. What about the other examples of the ToM task (the agent writing the false label themselves, or having been told by a trusted friend what is actually in the bag)?

Reply