Thanks, gwern, this is extremely helpful.

I think you're overthinking it.

This is quite likely. :)

In all the n-back research I've read, hardly anyone seems to've cared about how exactly the adaptiveness was implemented...

The adaptiveness (in Brain Workshop, which is the only implementation I've spent much time with) feels pretty horrible. At least compared to a typical modern well-balanced game.

Play a bunch of level with random settings for each parameter, then regress the scores on all the parameters. There, now you have a predictor.

You're talking about least squares, or some modification of least squares that deals with outliers, right? Doesn't this assume linearity? Or I suppose I'd have to model, say, the effect of increasing the N-back as aN^b (or some other suitable formula) and plug in values for a and b as parameters? But that means I have to take some time choosing a good model.

Also, it's important for users to have a good first experience, so I'd like to require as little personalized training data as possible (I will have some data available, since the adaptive mode won't be unlocked until clearing some number of earlier baked-in stages). Having some notion of a standard difficulty-with-respect-to-an-idealized-player should help with this.

To be fair, I haven't actually tried running any kind of regression like this yet, and this approach pretty clearly seems worth trying.

So, a log model.

Log in the number of plays, I assume. This sounds pretty reasonable, but I expect won't take into account some of the more interesting features of each player's skill trajectory, especially early on. Of course this could be based on my misinterpretations of personal experiences with learning, so I could easily be wrong. It would certainly make things simpler!

Why didn't any simpler approaches work?

If you mean simple like least squares regression, mostly because I didn't try it. If you mean a simple formula, I wouldn't be surprised if none exists (the formula I'm using is actually not quite as complicated as I made it sound). Anyway, my understanding is that even if I use regression on individual user data I'd need to use a pretty complex model with lots of parameters to make it work. Is this not true?

Also, is the Bernoulli trials stuff reasonable, or am I making things too complicated with that too?