LESSWRONG
LW

All of ReaderM's Comments + Replies

Modern Transformers are AGI, and Human-Level
ReaderM1-2

Not really. The majority of your experiences and interactions are forgotten and discarded, the few that aren't are recalled and triggered by the right input when necessary and not just sitting there in your awareness at all times. Those memories are also modified at every recall.

And that's really just beside the point. However you want to spin it, evaluating that many positions is not necessary for backtracking or playing chess. If that's the base of your "impossible" rhetoric then it's a poor one.

Reply
Modern Transformers are AGI, and Human-Level
ReaderM12

You can call it a "gut claim" if that makes you feel better.  But the actual reason is I did some very simple math (about the window size required and given quadratic scaling for transformers) and concluded that practically speaking it was impossible.

If you're talking about this:

Now imagine trying to implement a serious backtracking algorithm.  Stockfish checks millions of positions per turn of play.  The attention window for your "backtracking transformer" is going to have to be at lease {size of chess board state}*{number of positions eval

... (read more)
Reply
2Logan Zoellner
  Humans are not transformers. The "context window" for a human is literally their entire life.
Modern Transformers are AGI, and Human-Level
ReaderM12

Have you never figured out something by yourself?  The way I learned to do Sudoku was: I was given a book of Sudoku puzzles and told "have fun".

So few shot + scratchpad ?

I didn't say it was impossible to train an LLM to play Chess. I said it was impossible for an LLM to teach itself to play a game of similar difficulty to chess if that game is not in it's training data.

More gut claims. 

What they do not do is teach themselves things that aren't in their training data via trial-and-error.  Which is the primary way humans learn things

Setting up... (read more)

Reply
2Logan Zoellner
  I agree.  Or at least, I don't see any reason why not. My point was not that "a relatively simple architecture that contains a Transformer as the core" cannot solve problems via trial and error (in fact I think it's likely such an architecture exists).  My point was that transformers alone cannot do so. You can call it a "gut claim" if that makes you feel better.  But the actual reason is I did some very simple math (about the window size required and given quadratic scaling for transformers) and concluded that practically speaking it was impossible. Also, importantly, we don't know what that "relatively simple" architecture looks like.  If you look at the various efforts to "extend" transformers to general learning machines, there are a bunch of different approaches: alpha-geometry, diffusion transformers, baby-agi, voyager, dreamer, chain-of-thought, RAG, continuous fine-tuning, V-JEPA.  Practically speaking, we have no idea which of these techniques is the "correct" one (if any of them are). In my opinion saying "Transformers are AGI" is a bit like saying "Deep learning is AGI".  While it is extremely possible that an architecture that heavily relies on Transformers and is AGI exists, we don't actually know what that architecture is. Personally, my bet is either on a sort of generalized alpha-geometry approach (where the transformer generates hypothesis and then GOFAI is used to evaluate them) or Diffusion Transformers (where we iteratively de-noise a solution to a problem).  But I wouldn't be at all surprised if a few years from now it is universally agreed that some key insight we're currently missing marks the dividing line between Transformers and AGI.
Modern Transformers are AGI, and Human-Level
ReaderM32

sure.  4000 words (~8000 tokens) to do a 9-state 9-turn game with the entire strategy written out by a human.

Ok? That's how you teach anybody anything. 

Now extrapolate that to chess, go, or any serious game.

LLMs can play chess, poker just fine. gpt 3.5-turbo-instruct plays at about 1800 Elo, consistently making legal moves. - https://github.com/adamkarvonen/chess_gpt_eval

Then there is this grandmaster level chess transformer - https://arxiv.org/abs/2402.04494

Poker - https://arxiv.org/abs/2308.12466

And this doesn't address at all my actual point,

... (read more)
Reply
2Logan Zoellner
  Have you never figured out something by yourself?  The way I learned to do Sudoku was: I was given a book of Sudoku puzzles and told "have fun". I didn't say it was impossible to train an LLM to play Chess. I said it was impossible for an LLM to teach itself to play a game of similar difficulty to chess if that game is not in it's training data. These are two wildly different things. Obviously LLMs can learn things that are in their training data.  That's what they do.  Obviously if you give LLMs detailed step-by-step instructions for a procedure that is small enough to fit in its attention window, LLMs can follow that procedure.  Again, that is what LLMs do. What they do not do is teach themselves things that aren't in their training data via trial-and-error.  Which is the primary way humans learn things.
Modern Transformers are AGI, and Human-Level
ReaderM1-2

GPT-4 can play tic-tac-toe 

https://chat.openai.com/share/75758e5e-d228-420f-9138-7bff47f2e12d

Reply
2Logan Zoellner
sure.  4000 words (~8000 tokens) to do a 9-state 9-turn game with the entire strategy written out by a human.  Now extrapolate that to chess, go, or any serious game. And this doesn't address at all my actual point, which is that Transformers cannot teach themselves to play a game.
On Board Vision, Hollow Words, and the End of the World
ReaderM00

Not sure what you mean by 100 percent accuracy and of course, you probably already know this but 3.5 Instruct Turbo plays chess at about 1800 ELO fulfilling your constraints (and has about 5 illegal moves (potentially less) in 8205) https://github.com/adamkarvonen/chess_gpt_eval

Reply
The idea that ChatGPT is simply “predicting” the next word is, at best, misleading
ReaderM10

They can compute a state prior to each generated token and they can choose a token that signal a preservation of this state.

Reply
Large Language Models can Strategically Deceive their Users when Put Under Pressure.
ReaderM30

They had access to and tested the base un-RLHF'd model. Doesn't change much. RLHF has slightly higher misalignment and deception rates(which is a bit notable) but otherwise similar behavior.

Reply
2the gears to ascension
Huh.
AI Timelines
ReaderM40

Optimal tic tac toe takes explaining the game in excruciating detail. https://chat.openai.com/share/75758e5e-d228-420f-9138-7bff47f2e12d

Reply
1clydeiii
Skip over tic-tac-toe and go straight to chess: 
AI Timelines
ReaderM70

Optimal play requires explaining the game in detail. See here

https://chat.openai.com/share/75758e5e-d228-420f-9138-7bff47f2e12d

Reply
9gwern
More specifically: https://arxiv.org/pdf/2303.08774.pdf#page=12 ---------------------------------------- Dynomight, are you aware that, in addition to the GPT-4 paper reporting the RLHF'd GPT-4 being badly de-calibrated, there's several papers already examining the calibration and ability of LLMs to forecast?
Alignment Implications of LLM Successes: a Debate in One Act
ReaderM*10

I don't understand your position. Are you saying that if we generated protein sequences by uniformly randomly independently picking letters from "ILVFMCAGPTSYWQNHEDKR" to sample strings, and then trained an LLM to predict those uniform random strings, it would end up with internal structure representing how biology works? Because that's obviously wrong to me and I don't see why you'd believe it.

Ah no. I misunderstood you here. You're right.

**What I was trying to get at the notion that something in particular (Human, evolution etc) has to have "figured s... (read more)

Reply1
4tailcalled
I don't believe your idea that neural networks are gonna gain superhuman ability through inverting hashes because I don't believe neural networks can learn to invert cryptographic hashes. They are specifically designed to be non-invertible.
Alignment Implications of LLM Successes: a Debate in One Act
ReaderM-1-2

>They find functions that fit the results. Most such functions are simple and therefore generalize well. But that doesn't mean they generalize arbitrarily well.

You have no idea how simple the functions they are learning are. 

>Not really any different from the human language LLM, it's just trained on stuff evolution has figured out rather than stuff humans have figured out. This wouldn't work if you used random protein sequences instead of evolved ones.

It would work just fine. The model would predict random arbitrary sequences and the structure w... (read more)

Reply
4tailcalled
I don't understand your position. Are you saying that if we generated protein sequences by uniformly randomly independently picking letters from "ILVFMCAGPTSYWQNHEDKR" to sample strings, and then trained an LLM to predict those uniform random strings, it would end up with internal structure representing how biology works? Because that's obviously wrong to me and I don't see why you'd believe it. The algorithm that uses a Fourier transform for modular multiplication is really simple. It is probably the most straightforward way to solve the problem with the tools available to the network, and it is strongly related to our best known algorithms for multiplication. My claim is that for our richest data, the causal processes that inform the data is human intelligence. Of course you are right that there are other datasets available, but they are less rich but sometimes useful (as in the case of proteins). Furthermore what I'm saying is that if the AI learns to create its own information instead of relying on copying data, it could achieve much more. Plausibly true, but don't our best game-playing AIs also do self-play to create new game-playing information instead of purely relying on other's games? Like AlphaStar.
Alignment Implications of LLM Successes: a Debate in One Act
ReaderM21

Large language models gain their capabilities from self-supervised learning on humans performing activities, or from reinforcement learning from human feedback about how to achieve things, or from internalizing its human-approved knowledge into its motivation. In all of these cases, you rely on humans figuring out how to do stuff, in order to make the AI able to do stuff, so it is of course logical that this would tightly integrated capabilities and alignment in the way Simplicia says.

No. Language Models aren't relying on humans figuring anything out. How ... (read more)

Reply
2tailcalled
They find functions that fit the results. Most such functions are simple and therefore generalize well. But that doesn't mean they generalize arbitrarily well. Not really any different from the human language LLM, it's just trained on stuff evolution has figured out rather than stuff humans have figured out. This wouldn't work if you used random protein sequences instead of evolved ones. They try to predict the results. This leads to predicting the computation that led to the results, because the computation is well-approximated by a simple function and they are also likely to pick a simple function. Inverting relationships like this is a pretty good use-case for language models. But here you're still relying on having an evolutionary ecology to give you lots of examples of proteins.