Self-motivated hard work is the primary source of the intense, optimistic engagement known as flow—one of the greatest forms of happiness that makes us come alive with purpose and potential (Csikszentmihalyi, 1975). Sadly, for most people work does not feel so rewarding most of the time. Instead we often have to persevere through long periods of hard, painful, and unrewarding work when we could be doing something much more enjoyable. When faced with this motivational challenge people often give up too easily, get sidetracked, or procrastinate (Steel, 2007). The problem is not that we are not willing or unable to work hard. To the contrary, we crave being productively engaged in challenging tasks. Thus, instead of blaming ourselves for our limited will-power, it may be more productive to take a critical look at the carrots and the sticks that are supposed to help us stay motivated. Who put them there and why? Are these incentives helpful, distracting, irrelevant, or out of sight? If you could place them differently and add new ones, where would they go? Often, the problem is that the rewards we experience in the short run are misaligned with what we want to accomplish. In the short run the extremely valuable work that brings us closer to our cherished goals can be aversive while activities that are irrelevant or even opposed to everything we want to accomplish can be pleasant and rewarding. Hence, when we struggle to be engaged with something that we care about, then perhaps we are not the problem but the incentives are, or as Jane McGonigal (2011) put it "Reality is broken".

So, if reality is broken, then what can we do to fix it?  One approach is to design better incentive structures that make the pursuit of our goals more engaging. If we want to go this way, then there is a lot to be learned from games, because their incentive structures are so well designed that they let people enjoy hard work for many hours on end (McGonigal, 2011). In the past five years, the success of video games has inspired the gamification of education, work, health, and business. Gamification is the use of game elements, like points, levels, badges, and quests to engage, motivate, and nudge people in non-game contexts. There are even tools like SuperBetter and Habitica that individuals like you and I can use to gamify our own lives. Previous studies have shown that gamification can have positive effects on motivation, engagement, behavior, learning outcomes, and health—but only when it is done right (Hamari, Koivisto, & Sarsa, 2014; Roepke, et al., 2015). But when gamification is done wrong it can have negative effects by incentivizing counter-productive behaviors. So far gamification has been an art, and there is very little science about how to do it right. This motivated my advisor and me to develop a practical theory of optimal gamification.

In this blog post I focus on how our theory could be applied in practice. If you would like to learn about the technical details or read more about our experiments, then please take look at our CogSci paper (Lieder & Griffiths, submitted). I will start with a very brief summary of our method, provide an intuitive explanation of what it does, and then dive into how you can implement it in your own life. I will close with an outlook on how our method could be applied to gamify our todo lists.

Level 1: Optimal Gamification

Our method for optimal gamification draws on the theory of Markov decision processes (MDPs; Sutton & Barto, 1998) and the shaping theorem (Ng, Harada, & Russell, 1999). The basic idea is to align each action's immediate reward with its value in the long run. Therefore the points should complement the immediate rewards of doing something (e.g., how painful it is) by the value that it generates in the long run. Concretely, the points awarded for an activity should be chosen such that the right thing to do looks best in the short run when you combine how many points it is worth with how it feels when you do it. Furthermore, the points have to be assigned in such a way that when you undo something you lose as many points as you earned when you did it. We evaluated the effectiveness of our method in two behavioral experiments. Our first experiment demonstrated that incentive structures designed by our method can indeed help people make better, less short-sighted decisions—especially when course of action that is best in the long run is unpleasant in the short run. We also found that less principled approaches to gamification can encourage ruthless rushing towards a goal that causes more harm than good, and we showed that our method is guaranteed to avoid these perils. In the second experiment we found that the optimal incentive structures designed with our method can be effectively implemented using game elements like points and badges. These results suggest that the proposed method provides a principled way to leverage gamification to help people make better decisions.

Our method proceeds in three steps:

1.    Model the situation and the decision-maker's goals and options as a MDP.

2.    Solve the MDP to obtain the optimal value function V* or approximate it.

3.    Set the number of points for progressing from stage s to stage s' to V*(s')-V*(s).  

Intuitively, this means that the number of points that is awarded for doing something should reflect how much better the resulting state (i.e., s') is than the previous one (i.e., s). For instance, achieving a goal is worth 1000 points then completing 10% of the work required to reach the goal should be rewarded with 100 points. So let's think about how you could apply this approach right now without having to solve MDPs.

Level 2: Practical Implications

In my day-to-day life I try to approximate optimal gamification as follows:

1.    Set a concrete goal that you would like to achieve and figure out how many points it is worth, e.g. writing this blog post was worth 1000 points to me.

2.    Set several milestones along the way to the goal to divide the path into small steps that feel very manageable.

3.    For each milestone, determine how far you will have come when you get there as a percentage of the total distance to the goal, e.g. 10%, 20%, 30%, ..., 100% for the first, second, third, ..., and the tenth milestone respectively.

4.    Assign each milestone the corresponding fraction of the total value of achieving the goal, e.g. 100 points, 200 points, 300 points, ..., and 1000 points for the first, second, third, ..., and tenth milestone respectively.

5.    Figure out what you have to do to get from one milestone to the next. If this is a simple activity, then its reward should be the difference between the value of next milestone and the value of the current milestone, e.g. 100 points. If it is a complex sequence of actions, then make it a subgoal and apply steps 1-3 figure out how to achieve it.

6.    Once you are done with step 5, you can add those points to your todo-list.

7.    Now it is time to get things done and reward yourself. You start at 0 points, but whenever you complete one of the steps, you earn as many points as you have assigned to it and can increment your (daily) score.

Earning these points can be very rewarding if you remind yourself what they stand for. If your goal was worth $1,000,000 to you and you assigned 1000 points to it, then 10 points should be worth $10,000 to you. But if this is not rewarding enough for you, you can think of ways that make the points more pleasurable. You could, for instance, make a high-score list that motivates you to beat your personal best day after day or start a high-score competition with your friends. You could also set yourself the goal to achieve a certain number of points by a certain time and promise yourself a treat if you achieve it.

There are many other ways that you could assign points to the items on you todo list. Feel free to do whatever works for you. But it may be useful to keep in mind that the way in which optimal gamification assigns points has several formal properties that are necessary to avoid negative side-effects:

a) Each item's score reflects how valuable is in the long run.

Optimal gamification works because it aligns each action's immediate reward with its long-term value. To help you make better decisions the points should be designed such that the course of action that is best in the long run looks best in the short run. This entails incentivizing unpleasant or unrewarding activities that will pay off later—especially when their less productive alternatives are very rewarding in the short run.

b) Beware of cycles!

The shaping theorem (Ng, et al., 1999) requires that going back and forth between two states receives a net pseudo-reward of zero. When your pseudo-rewards along a circle add up to a positive value, then you may be incentivizing yourself to create unnecessary problems for yourself. This can happen when the action for which you reward yourself can only be executed in an undesirable state, and you do not equally punish yourself for falling back into that state. For instance adding points for losing weight will inadvertently incentivize you to regain weight afterwards unless you subtract at least the same number of points for gaining weight. Similarly, if you reward yourself for solving interpersonal conflicts but don’t punish yourself for creating them, then you may be setting yourself up for trouble. To avoid such problems, creating a problem must be punished by at least as many points as you earn by solving it. 

c) Two ways to achieve the same goal should yield the same number of points.

The shaping theorem also requires that all paths that lead to the same final state (e.g., having submitted a paper by the deadline) should yield the same amount of reward. If this is not the case your pseudo-rewards may bias you towards a suboptimal path. For instance, if you reward your all-nighter on the last night before the deadline by the reward value of a month’s worth of work, you are incentivizing yourself to procrastinate. Similarly, if you reward one activity that leads towards your goal much more heavily than others, then you may be biasing yourself towards a reckless course of action that may achieve the goal at an unreasonably high cost. For instance, rewarding yourself 100 times as much for working 100% on a project than for working on it 50% might lead you to complete the project early at the expense of your health, your friendships, your education, and all your other projects. To avoid this problem, al paths that lead to the same state should yield the same amount of reward. 

d) Pseudo-rewards should be awarded for state-transitions instead of actions.

Many applications of gamification reward "good" actions with points regardless of when or how often these actions are taken. But according to the shaping theorem, the number of points must depend on the state in which the action is taken and the state that it leads to. If your pseudo-rewards were based only on what you do but not on when you do it, then you might keep rewarding yourself for something even when it is no longer valuable, because the underlying state has changed. For instance, at some point your reward for losing weight has to diminish or else you may be setting yourself up for anorexia.

Level 3: Todo-list gamification

My first practical application is to manually gamify my todo-list every morning. I find this very helpful and motivating: Assigning points to the items on my todo list makes me realize how much I value them. This is useful for prioritizing important task. Earning points allows me to perceive my progress more more accurately and more vividly. This helps me feel great about getting something important done even when it was only a single item on my todo list and took me a lot of time and effort to accomplish. Conversely, the point scheme also prevents me from feeling so good about checking off small things that I become tempted to neglect the big ones that are much more important. Gamification thereby remedies the todo list's shortcoming that it makes each item seem equally important. I highly recommend gamifying your todo lists. It can be highly motivating. Yet, adding the points manually takes some effort and my point scheme is often somewhat arbitrary and probably suboptimal.

To make todo list gamification easier and more effective, I am planning to develop an easy-to-use website or app that will do optimal gamification for you. Its graphical user interface would allow you to create hierarchical todo-lists, ask you 1 or 2 simple questions about each item on your list and then gamify your todo-list for you. To do this, it will translate your list and your answers into a MDP, compute its optimal value function, and use it to determine how valuable it is to complete each item. The tool could also help you set manageable subgoals and determine what is most important and should be done first. Last but not least, a website or app can also leverage additional game elements to make the points that you earn more rewarding: It can track your productivity and provide instant feedback that makes your progress more salient. It can send you on a quest that gives you a goals along with small actionable steps. The tool could allow you to realize that you are getting ever more productive by visualizing your progress over time. As you become more effective, you level up and your quests will become increasingly more challenging.  It might include a scoreboard that lets you compete with yourself and/or others and win prizes for your performance. Last but not least, if you need an extra push, you can tie your points to social rewards, your favorite treat, money, or access to your favorite music, apps, or websites. There are many more possibilities, and I invite you to think about it and share your ideas. In brief, there is wealth of opportunities to leverage game elements to make goal achievement fun and easy.

Join me on my quest! An adventure awaits. 

Gamification can be a useful tool to make achieving your goals easier and more engaging. However, gamification only works when it is done right. The theory of MDPs and pseudo-rewards provide the formal tools needed to do gamification right. With the help of these tools we can design incentive structures that help people overcome motivational obstacles, do the right thing and achieve their goals. But more research and development needs to be done to make optimal gamification practical.

If you have any thoughts or ideas for what to do next, noticed a problem with the approach, or would like to be part of our team and contribute to building a tool helps people achieve their goals, please send me an e-mail.

References and recommended readings

Csikszentmihalyi, M. (1975). Beyond boredom and anxiety: the experience of play in work and games. San Francisco: Jossey-Bass.

Lieder, F., & Griffiths, T.L. (submitted). Helping people make better decisions using optimal gamification. CogSci 2016. [Manuscript]

McGonigal, J. (2011). Reality is broken: Why games make us better and how they can change the world. New York: Penguin.

McGonigal, J. (2015). SuperBetter: A revolutionary approach to getting stronger, happier, braver and more resilient–powered by the science of games. London, UK: Penguin Press.

Hamari, J., Koivisto, J., & Sarsa, H. (2014). Does gamification work?–A literature review of empirical studies on gamification. In 47th Hawaii international conference on system sciences (pp. 3025–3034).

Ng, A. Y., Harada, D., & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. In I. Bratko & S. Dzeroski (Eds.), Proceedings of the 16th annual international conference on machine learning (Vol. 16, pp. 278–287). San Francisco, CA, USA: Morgan Kaufmann.

Roepke, A. M., Jaffee, S. R., Riffle, O. M., McGonigal, J., Broome, R., & Maxwell, B. (2015). Randomized controlled trial of SuperBetter, a smartphone-based/Internet-based self-help tool to reduce depressive symptoms. Games for health journal, 4(3), 235-246.

Sutton, R. S., & Barto, A. G. (1998). Reinforcement learning: An introduction. Cambridge, MA, USA: MIT press.

 

While making the inventory of my personal library and applying the Universal Decimal System to its classification, I found myself discovering a systematized classification of fields of knowledge, nested and organized and intricate, many of which I didn't even know existed. I couldn't help but compare how information was therein classified, and how it was imparted to me in engineering school. I also thought about how, often, software engineers and computer scientists were mostly self-thought, with even college mostly consisting of "here's a problem: go forth and figure out a way to solve it". This made me wonder whether another way of certified and certifiable education couldn't be achieved, and a couple of ideas sort of came to me.

It's pretty nebulous in my mind so far, but the crux of the concept would be a modular structure of education, where the academic institution essentially established what information precisely you need from each module, and lets you get on with the activity of learning, with periodic exams that you can sign up for, which will certify your level and area of proficiency in each module.

A recommended tree of learning can be established, but it should be possible to not take every intermediate test, if passing the final test proves that you've passed all the others behind it (this would allow people coming from different academic systems to certify their knowledge quickly and easily, thus avoiding the classic "Doctor in Physics from Former Soviet Union, current Taxi Driver in New York" scenario).

Thus, a universal standard of how much you have proven to know about what topics can be established.

Employers would then be free to request profiles in the format of such a tree. It need not be a binary "you need to have done all these courses and only these courses to work for us", they could be free to write their utility function for this or that job however they would see fit, with whichever weights and restrictions they would need.

Students and other learners would be free to advance in whichever tree they required, depending on what kind of profile they want to end up with at what age or point in time. One would determine what to learn based on statistical studies of what elements are, by and large, most desired by employers of/predictors of professional success in a certain field you want to work in.

One would find, for example, that mastering the peculiar field of railway engineering is great to be a proficient railway engineer, but also that having studied, say, things involved with people skills (from rhetoric to psychology to management), correlates positively with success in that field.

Conversely, a painter may find that learning about statistics, market predictions, web design, or cognitive biases correlates with a more successful career (whether it be on terms of income, or in terms of copies sold, or of public exposure... each one may optimize their own learning according to their own criteria).

One might even be able to calculate whether such complimentary education is actually worth their time, and which of them are the most cost-efficient.

I would predict that such a system would help society overall optimize how many people know what skills, and facilitate the learning of new skills and the updating of old ones for everyone, thus reducing structural unemployment, and preventing pigeonholing and other forms of professional arthritis.

I would even dare to predict that, given the vague, statistical, cluster-ish nature of this system, people would be encouraged to learn quite a lot more, and on a quite wider range of fields, than they do now, when one must jump through a great many hoops and endure a great many constraints in space and time and coin to get access to some types of educations (and to the acknowledgement of their acquisition thereof).

Acquiring access to the actual sources of knowledge, a library (virtual or otherwise), lectures (virtual or otherwise), and so on, would be a private matter, up to the learner:

• some of them already have the knowledge and just need to get it certified,
• others can actually buy the books they want/need, especially if keeping them around as reference will be useful to them in the future,
• others can subscribe to one or many libraries, of the on-site sort or by correspondence
• others can buy access to pre-recorded lectures, peruse lectures that are available for free, or enroll in academic institutions whose ostensible purpose is to give lectures and/or otherwise guide students through learning, more or less closely
• the same applies to finding study groups with whom you can work on a topic together: I can easily imagine dedicated social networks could be created for that purpose, helping people pair up with each other based on mutual distance, predicted personal affinity, mutual goals, backgrounds, and so on. Who knows what amazing research teams might be borne of the intellectual equivalent of OK!Cupid.

A thing that I would like very much about this system is that it would free up the strange conflicts of interest that hamper the function of traditional educational institutions.

When the ones who teach you are also the ones who grade you, the effort they invest in you can feel like a zero-sum game, especially if they are only allowed to let a percentage of you pass.

When the ones who teach you have priorities other than teach (usually research, but some teachers are also involved in administrative functions, or even private interests completely outside of the university's ivory tower¹), this can and often does reduce the energy and dedication they can/will allocate to the actual function of teaching, as opposed to the others.

By separating these functions, and the contradictory incentives they provide, the organizations performing them are free to optimize for each: 

• Testing is optimized for predicting current and future competence in a subject: the testers whose tests are the most reliable have more employers requiring their certificates, and thus more people requesting that they test them
• Teaching is optimized for getting the knowledge through whatever the heck the students want, whether it be to succeed at the tests or to simply master the subject (I don't know much game theory, but I'd naively guess that the spontaneous equilibrium between the teaching and testing institutions would lead to both goals becoming identical).
• Researching is optimized for research (researchers are not teachers. dang it, those are very different skill-sets!). However researchers and other experts get to have a pretty big say in what the tests test for and how, because their involvement makes the tests more trustworthy for employers, and because they, too, are employers.
• And of course entire meta-institutions can spring from this, whose role is to statistically verify, over the long term,
  
  • how good a predictor of professional success in this or that field is passing the corresponding test, and
  • how good a predictor of passing the test is to be taught by this or that teaching institution.
  • how good a predictor of the test being reliable is the input of these or those researchers and experts
• It occurs to me now that, if one wished to be really nitpicky about who watches the watchmen, I suspect that there would be institutions testing the reliability of those meta-institutions, and so on and so forth... When does it stop? How to avoid vested interests and little cheats and manipulations pulling an academic equivalent of the AAA certification of sub-prime junk debt in 2008?

Another discrepancy I'd like to see solved is the difference between the official time it is supposed to take to obtain this or that degree, to learn this or that subject, and the actual statistical distribution of that time. Nowadays, a degree that's supposed to take you five years ends up taking up eight or ten years of your life. You find yourself having to go through the most difficult subjects again and again, because they are explained in an extremely rushed way, the materials crammed into a pre-formatted time. Other subjects are so exceedingly easy and thinly-spread that you find that going to class is a waste of time, and that you're better off preparing for it one week before finals. Now, after having written all of the above, my mind is quite spent, and I don't feel capable of either anticipating the effect of my proposed idea on this particular, nor of offering any solutions. Nevertheless, I wish to draw attention to this, so I'm leaving this paragraph in until I can amend it to something more useful/promising.

I hereby submit this idea to the LW community for screening and sound-boarding. I apologize in advance for your time, just in case this idea appears to be flawed enough to be unsalvageable. If you deem the concept good but flawed, we could perhaps work on ironing those kinks together. If, afterwards, this seems to you like a good enough idea to implement, know that good proposals are a dime a dozen; if there is any interest in seeing something like this happen, we can need to move on to proprely understanding the current state of secondary/superior/higher education, and figuring out of what incentives/powers/leverages are needed to actually get it implemented.

 

---

---

 

¹By ivory tower I simply mean the protected environment where professors teach, researchers research, and students study, with multiple buffers between it and the ebb and flow of political, economical, and social turmoil. No value judgement is intended.

 

---

 

EDIT: And now I look upon the title of this article and realize that, though I had comparisons to games in mind, I never got around to writing them down. My inspirations here were mostly Civilization's Research Trees, RPG Skill Scores and Perks, and, in particular, Skyrim's skills and perks tree.

Basically, your level at whatever skill improves by studying and by practising it rather than merely by levelling up, and, when you need to perform a task that's outside your profile, you can go and learn it without having to commit to a class. Knowing the right combination of skills at the right level lets you unlock perks or access previously-unavailable skills and applications. What I like the most about it is that there's a lot of freedom to learn what you want and be who you want to be according to your own tastes and wishes, but, overall, it sounds sensible and is relatively well-balanced. And of course there's the fact that it allows you to keep a careful tally of how good you are at what things, and the sense of accomplishment is so motivating and encouraging!

Speaking of which, several netwroks and consoles' Achievement systems also strike me as motivators for keeping track of what one has achieved so far, to look back and be able to say "I've come a long way" (in an effect similar to that of gratitude journals), and also to accomplish a task and have this immediate and universal acknowledgement that you did it dammit (and, for those who care about that kind of thing, the chance to rub it the face of those who haven't).

I would think our educational systems could benefit from this kind of modularity and from this ability to keep track of things in a systematic way. What do you guys think?