Cross-posted to my blog. I expect this will be of some interest to the LessWrong community both because of previous interest in N-back and because of the opportunity to apply Bayesian statistics to a real-world problem. The main reason I'm writing this article is to get feedback on my approach and to ask for help in the areas where I'm stuck. For some background, I'm a software developer who's been working in games for 7+ years and recently left my corporate job to work on this project full-time.

As I mentioned here and here, since early February I've been working on an N-back-like mobile game. I plan to release for iOS this summer and for Android a few months later if all goes well. I have fully implemented the core gameplay and most of the visual styling and UI, and am currently working with a composer on the sound and music.

I am just now starting on the final component of the game: an adaptive mode that assesses the player's skill and presents challenges that are tuned to induce a state of flow.

The Problem

The game is broken down into waves, each of which presents an N-back-like task with certain parameters, such as the number of attributes, the number of variants in each attribute, the tempo, and so on. I would like to find a way to collapse these parameters into a single difficulty parameter that I can compare against a player's skill level to predict their performance on a given wave.

But I realize that some players will be better at some challenges than others (e.g. memory, matching multiple attributes, handling fast tempos, dealing with visual distractions like rotation, or recognizing letters). Skill and difficulty are multidimensional quantities, and this makes performance hard to predict. The question is, is there a single-parameter approximation that delivers an adequate experience? Additionally, the task is not pure N-back — I've made it more game-like — and as a result the relationship between the game parameters and the overall difficulty is not as straightforward as it would be in a cleaner environment (e.g. difficulty might be nearly linear in tempo for some set-ups but highly non-linear for others).

I have the luxury of having access to fairly rich behavioral data. The game is partly a rhythm game, so not only do I know whether a match has been made correctly (or a non-match correctly skipped) but I also know the timing of a player's positive responses. A player with higher skill should have smaller timing errors, so a well-timed match is evidence for higher skill. I am still unsure exactly how I can use this information optimally.

I plan to display a plot of player skill over time, but this opens another set of questions. What exactly am I plotting? How do I model player skill over time (just a time-weighted average? as a series of slopes and plateaus? how should I expect skill to change over a period of time without any play?)? How much variation in performance is due to fatigue, attention, caffeine, etc.? Do I show error bars or box plots? What units do I use?

And finally, how do I turn a difficulty and a skill level into a prediction of performance? What is the model of the player playing the game?

Main Questions

• Is there an adequate difficulty parameter and if so how do I calculate it?
• Can I use timing data to improve predictions? How?
• What model do I use for player skill changing over time?
• How do I communicate performance stats to the user? Box and whiskers? Units?
• What is the model of the player and how do I turn that into a prediction?

My Approach

I've read Sivia, so I have some theoretical background on how to solve this kind of problem, but only limited real-world experience. These are my thoughts so far.

Modeling gameplay performance as Bernoulli trials seems ok. That is, given a skill level S and a difficulty D, performance on a set of N matches should be closely matched by N Bernoulli trials with probability of success p(S, D) as follows:

• if S ≪ D, p = 0.5
• if S ≫ D, p is close to 1.0 (how close?)
• if S = D, p = 0.9 feels about right
• etc.

Then I can update S (and maybe D? see next paragraph) on actual player performance. This will result in a new probability density function over the "true" value of S, which will hopefully be unimodal and narrow enough to report as a single best estimate (possibly with error bars). Which reminds me, what do I use as a prior for S? And what happens if the player just stops playing halfway through, or hands the game to their 5-year-old?

Determining difficulty is another hard problem. I currently have a complicated ad-hoc formula that I cobbled together with logarithms, exponentials, and magic numbers, and lots of trial and error. It seems to work pretty well for the limited set of levels I've tested with a small group of playtesters, but I'm worried that it won't predict difficulty well outside of that domain. One possibility is to croud-source it: after release I'd collect performance data across all users and update the difficulty ratings on the fly. This seems risky and difficult, and the initial difficulty ratings might be way off, which would lead to poor initial user experiences with the adaptive mode. I would also have to worry about maintaining a server back-end to gather the data and report on updated difficulty levels.

Request For Feedback

So, any suggestions on how to tackle these problems? Or the first place to start looking?

I'm pretty excited about the potential to collect real-world data on skill acquisition over time. If there is sufficient interest I'll consider making the raw data public, and even instrument the code to collect other data of interest, by request. I do have some concerns over data privacy, so I may allow users to opt out of sending their data up to the server.

As Eliezer makes the point that real utopias will be scary - certainly more scary that my latest attempt. Mainly they will be scary because they'll be different, and humans don't like different, and it's vital that the authors realise this if they want to create a realistic scenario. It's necessary to craft a world where we would be out of place.

But it's important to remember that utopias will not be scary for the people living there - the aspects that we find scary at the beginning of the 21st century are not what the locals will be afraid of (put your hand up if you are currently terrified that the majority of women can vote in modern democracies). Scary is in the observer, not the territory.

This is a special challenge when writing a fictional utopia. Dystopias and flawed utopias are much easier to write than utopias; when you can drop an anvil on your protagonist whenever you feel like it, then the tension and interest are much easier to sustain. And the scary parts of utopia are a cheap and easy way of dropping anvils: the reader thrills to this frightening and interesting concept, start objecting/agreeing/thinking about and with it. But it's all ok, you think, it's not dystopia, it's just a scary utopia; you can get your thrills without going astray.

But all that detracts from your real mission, which is to write a utopia that is genuinely good for the people in it, and would be genuinely interesting to read about even if it weren't scary. I found this particularly hard, and I'd recommend that those who write utopias do a first draft or summary without any scary bits in it - if this doesn't feel interesting on its own, then you've failed.

Then when you do add the scary bits, make sure they don't suck all the energy out of your story, and make sure you emphasise that the protagonists find these aspects commonplace rather than frightening. There is a length issue - if your story is long, you can afford to put more scary bits in, and even make the reader start seeing them just as the locals do, without the main point being swallowed up. If your story's short, however, I'd cut down on the scary radically: if "rape is legal" and you only have a few pages, then that's what most people are going to remember about your story. The scariness is a flavouring, not the main dish.

My grandfather died several years ago, before I began to seriously consider cryonics. He deteriorated markedly as he approached death. Nevertheless, he was smart enough to want to be part of the new technologies "the kids were putting out these days." At age 90, with some help, he created a blog and posted this entry:

As my memory weakens, I no longer perform much, but I deeply enjoy life. My wife, Genie, and my three daughters help me a lot, with food, walks, talks, and gifts. I usually feel good. Many other people say and do nice things for me.

...

Life began as cells 3 ½ billion years ago and gradually spread out from one species to another. There is no evidence of any species living after death. Therefore, each human should enjoy life, itself. Long before our Earth formed, our Universe spread out 14 billion years ago, long after material existed, which may have been forever. On that basis, I think human lives are a result of amazing development. We can enjoy life deeply into old age and on almost to death. No human should weaken true enjoyment by physically attacking another human. A human may argue with another human with the purpose of keeping both lives enjoyable.

 

When approaching death, a person should overcome huge pain by mental concentration or medicine and enjoy the remainder of life. This can be done by listening to music, relatives, friends, reading, and other actions. Life can be pleasant to the end, or about to the end. Enjoy yourself. Be nice to others.

 

The unedited entry (still up, along with other postings to his blog) concerns the mental topics that were occupying him at the end of life: mainly birdwatching and overpopulation. I post it here as evidence that even a very old person, suffering the mental and physical burdens of advanced old age, can still enjoy themselves and value life.

From one perspective, it is the ramblings of an old man succumbing to dementia. From another, it is proof that life is never a thing to be easily surrendered.

Enjoy yourself. Be nice to others.

This post is a formal request for everybody in this forum, who are the most likely humans to produce an FAI in the future, that in the possible resulting utopia some activity will still be available, even if they are not based on challenges of increasing complexity. I call these activity natural wireheadings, and since it is my right to determine my own fun-generating activities, I hereby formally request that some simple pleasure, listened below, will still be available at *any* point in my future cone, and that I will consider a dystopia any future which deprives me of these natural wireheadings (if anyone is still around caring for those things).
A (non-exhaustive) list of them includes the following:
- sex
- eating food/drinking
- feeling relieved for having emptied my bowels
- dancing
- the pleasure of physical activity (wood-carving, sculpting, running, etc)
- the rapture I feel when in the presence of a safe ancestral environment
- social laughter
- the pleasure of talking in a small, same-minded crowd
- listening to pop/metal/rap music
- the pleasure of resting when tired
- scratching an itch
-...

More will come!