When I say Pokémon-type games, I don’t mean games recounting the adventures of Ash Ketchum and Pikachu.  I mean games with a series of obstacles set in a large semi-open world, with things you can carry, a small set of available actions at each point, and a goal of progressing past the obstacles.  Such games can be manufactured in unlimited quantities by a program.  They can also be “peopled” by simple LLMs, for increased complexity.  They don’t actually have to be fun to play or look at, so the design requirements are loose.

There have been attempts at reinforcement learning using unlimited computer-generated games.  They haven’t worked that well.  I think the key feature that favors Pokémon-like games is that when the player dies or gets stuck, they can go back to the beginning and try again.  This rewards trial-and-error learning to get past obstacles, keeping a long-term memory, and to re-plan your approach when something doesn’t work.  These are capabilities in which current LLMs are notably lacking.

Another way of saying what Claude’s missing skill is: managing long-term memory.  You need to remember important stuff, forget minor stuff, summarize things, and realize when a conclusion in your memory is wrong and needs correction.

True.  I was generalizing it to a system that tries to solve lots of Pokémon-like tasks in various artificial worlds, rather than just expecting it to solve Pokémon over and over.  But I didn’t say that, I just imagined in my mind and assumed everyone else would too.  Thank you for making it explicit!

This is an important case to think about.  I think it is understudied.  What separates current AIs from the CEO role?  And how long will it take?  I see three things:

• Long term thinking, agency, the ability to remember things, not going crazy in an hour or two.  It seems to me like this is all the same problem, in the sense that I think one innovation will solve all of them.  This has a lot of effort focused on it.  I feel like it's been a known big problem since GPT-4 and Sydney/Bing, 2 1/2 years ago.  So, by the Lindy principle, it should be another 2 1/2 years until it is solved.
• Persuasiveness.  I've known a few CEOs in my life; they were all more persuasive than average, and one was genuinely unearthly in her ability to convince people of things.  LLMs have been steadily increasing in persuasiveness, and are now par-human.  So I think scaling will take care of this.  Perhaps a year or two?
• Experience.  I don't know how much of this can be inculcated with the training data, and how much of it requires actually managing people and products and thinking about what can go wrong. Every CEO has to deal with individual subordinates, customers, and counterparties.  How much of the job is learning the ins and outs of those particular people?  Or does it suffice to have spent a thousand subjective years reading about a million people?  So we may already have this.

(Side point: As an engineer watching CEOs, I am amazed by their ability to take a few scattered hints spread across their very busy days, and assemble them into a theory of what's going on and what to do about it.  They're willing to take what I would consider intolerably flimsy evidence and act on it.  When this doesn't work, its called "jumping to conclusions", when it does work it's called "a genius move".  AIs should be good at this if you turn up their temperature.)

Eyes are exactly the worst tissue in which to express luciferase.  Lighting the eyeball from within prevents seeing things outside the eye, causing blindness.  Make your cat glow anywhere else!

Epistemic status: I know you're kidding.  So am I.

My charity extends no further than the human race.  Once in a while I think about animal ethics and decide that no, I still don't care enough to make an effort.

A basic commitment of my charity group from the beginning: no money that benefits things other than people.  We don't donate to benefit political groups, organizations, arts, animals, or the natural world.  I'm good with that.  Members of the group may of course donate elsewhere, and generally do.

We've been doing this since 1998, decades before Effective Altruism was a thing.  I don't have a commitment to Effective Altruism the movement, just to altruism which is effective.

It seems to me that I have done a lot of careful thinking about timelines, and that I also feel the AGI.  Why can't you have a careful understanding what timelines we should expect, and also have an emotional reaction to that?  Reasonably coming to the conclusion that many things will change greatly in the next few years deserves a reaction.  

A task like this, at which the AI is lousy but not hopeless, is an excellent feedback signal for RL.  It's also an excellent feedback signal for "grad student descent": have a human add mechanisms, and see if Claude gets better.  This is a very good sign for capabilities, unfortunately.

You would suppose wrong!  My wife and I belong to a group of a couple of dozen people that investigates charities, picks the best ones, and sends them big checks. I used to participate more, but now I outsource all the effort to my wife.  I wasn’t contributing much to the choosing process.  I just earn the money 🙂.

What does this have to do with my camp#1 intuitions?

I don’t think enjoyment and suffering are arbitrary or unimportant.  But I do think they’re nebulous.  They don’t have crisp, definable, generally agreed-upon properties. We have to deal with them anyway.

I don’t reason about animal ethics; I just follow standard American cultural standards.  And I think ethics matters because it helps us live a virtuous and rewarding life.  

Is that helpful?

Well, I’m glad you’ve settled the nature of qualia.  There’s a discussion downthread, between TAG and Signer, which contains several thousand words of philosophical discussion of qualia.  What a pity they didn’t think to look in Wikipedia, which settles the question!

Seriously, I definitely have sensations.  I just think some people experience an extra thing on top of sensations, which they think is an indissoluble part of cognition, and which causes them to find some things intuitive that I find incomprehensible.