Benjamin_Todd

Gary Marcus now saying AI can't do things it can already do

January 2020, Gary Marcus wrote GPT-2 And The Nature Of Intelligence, demonstrating a bunch of easy problems that GPT-2 couldn’t get right. He concluded these were “a clear sign that it is time to consider investing in different approaches.” Two years later, GPT-3 could get most of these right. Marcus wrote a new list of 15 problems GPT-3 couldn’t solve, concluding “more data makes for a better, more fluent approximation to language; it does not make for trustworthy intelligence.” A year later, GPT-4 could get most of these right. Now he’s gone one step further, and criticised limitations that have already been overcome. Last week Marcus put a series of questions into chatGPT, found mistakes, and concluded AGI is an example of “the madness of crowds”. However, Marcus used the free version, which only includes GPT-4o. That was released in May 2024, an eternity behind the frontier in AI. More importantly, it’s not a reasoning model, which is where most of the recent progress has been. For the huge cost of $20 a month, I have access to GPT-o1 (not the most advanced model OpenAI offers, let alone the best that exists). I asked GPT-o1 the same questions Marcus did and it didn’t make any of the mistakes he spotted. First he asked it: > Make a table of every state in the US, including population, area and median household income, sorted in order of median household income. GPT-4o misses out a bunch of states. GPT-o1 lists all 50 (full transcript). Then he asked for a column added on population density. This also seemed to work fine. He then made a list of Canadian provinces and asked for a column listing how many vowels were in each name. I was running out of patience, so asked the same question about the US states. This also worked: To be clear, there are probably still some mistakes in the data (just as I’d expect from most human assistants). The point is that the errors Marcus identified aren’t showing up. He goes on to correctly point out that agent

62Feb 9, 2025

Benjamin_Todd

Message

347

14y

What can we learn from expert AGI forecasts?

An introductory article on AI forecasts. Posted for feedback. As a non-expert, it would be great if there were experts who could tell us when we should expect artificial general intelligence (AGI) to arrive. Unfortunately, there aren’t. There are only different groups of experts with different weaknesses. This article is...

Apr 9, 20255

The case for AGI by 2030

I spent a couple of weeks writing this new introduction to AI timelines. Posting here in case useful to share and for feedback. The aim is to be up-to-date, more balanced than Situational Awareness, but still relatively accessible. In recent months, the CEOs of leading AI companies have grown increasingly...

Apr 9, 202542

Teaching AI to reason: this year's most important story

This doesn't contain much new to LW readers. I wrote it to try to explain what's going on to a broader audience – I'm posting in case people find it helpful for that. Feedback welcome. Most people think of AI as a pattern-matching chatbot – good at writing emails, terrible...

Feb 13, 202510

Gary Marcus now saying AI can't do things it can already do

Feb 9, 202562

How quickly could robots scale up?

Once robots can do physical jobs, how quickly could they become a significant part of the work force? Here's a couple of fermi estimates, showing (among other things) that converting car factories might be able to produce 1 billion robots per year in under 5 years. Nothing too new here...

Jan 12, 202546

It looks like there are some good funding opportunities in AI safety right now

The AI safety community has grown rapidly since the ChatGPT wake-up call, but available funding doesn’t seem to have kept pace. However, there’s a more recent dynamic that’s created even better funding opportunities, which I witnessed as a recommender in the most recent SFF grant round.[1] Most philanthropic (vs. government...

Dec 22, 202420

How much AI inference can we do?

Suppose you have a bunch of GPUs. How many LLM forward passes can you do with them?[1] This is relevant to figuring out how profitable AI will be in the short-term, how powerful AI systems might be able to come in the near future, how large the compute overhang will...

May 14, 202420

Load More (7/9)

LESSWRONG
LW

LESSWRONG
LW

Benjamin_Todd

Benjamin_Todd

Benjamin_Todd

Gary Marcus now saying AI can't do things it can already do

How quickly could robots scale up?

The case for AGI by 2030

What are your questions about making a difference?

Benjamin_Todd

What can we learn from expert AGI forecasts?

The case for AGI by 2030

Teaching AI to reason: this year's most important story

Gary Marcus now saying AI can't do things it can already do

How quickly could robots scale up?

It looks like there are some good funding opportunities in AI safety right now

How much AI inference can we do?

Gary Marcus now saying AI can't do things it can already do

How quickly could robots scale up?

The case for AGI by 2030

What are your questions about making a difference?

What can we learn from expert AGI forecasts?

The case for AGI by 2030

Teaching AI to reason: this year's most important story

Gary Marcus now saying AI can't do things it can already do

How quickly could robots scale up?

It looks like there are some good funding opportunities in AI safety right now

How much AI inference can we do?