I sometimes go around saying that the fundamental question of rationality is Why do you believe what you believe?

-- Eliezer in Quantum Non-Realism

I was much impressed when they finally came out with a PC version of DragonBox, and I got around to testing it on some children I knew. Two kids, one of them four and the other eight years old, ended up blazing through several levels of solving first-degree equations while having a lot of fun doing so, even though they didn't know what it was that they were doing. That made me think that there has to be some way of making a computer game that would similarly teach rationality skills at the 5-second level. Some game where you would actually be forced to learn useful skills if you wanted to make progress.

After playing around with some ideas, I hit upon the notion of making a game centered around the Fundamental Question. I'm not sure whether this can be made to work, but it seems to have promise. The basic idea: you are required to figure out the solution to various mysteries by collecting various kinds of evidence. Some of the sources of evidence will be more reliable than others. In order to hit upon the correct solution, you need to consider where each piece of evidence came from, and whether you can rely on it.

Gameplay example

Now, let's go into a little more detail. Let's suppose that the game has a character called Bob. Bob tells you that tomorrow, eight o'clock, there will be an assassination attempt on Market Square. The fact that Bob has told you this is evidence for the claim being true, so the game automatically records the fact that you have such a piece of evidence, and that it came from Bob.

(Click on the pictures in case you don't see them properly.)

But how does Bob know that? You ask, and it turns out that Alice told him. So next, you go and ask Alice. Alice is confused and says that she never said anything about any assassination attempt: she just said that something big is going to be happen at the Market Square at that time, she heard it from the Mayor. The game records two new pieces of evidence: Alice's claim of something big happening at the Market Square tomorrow (which she heard from the Mayor), and her story of what she actually told Bob. Guess that Bob isn't a very reliable source of evidence: he has a tendency to come up with fancy invented details.

Or is he? After all, your sole knowledge about Bob being unreliable is that Alice claims she never said what Bob says she said. But maybe Alice has a grudge against Bob, and is intentionally out to make everyone disbelieve him. Maybe it's Alice who's unreliable. The evidence that you have is compatible with both hypotheses. At this point, you don't have enough information to decide between them, but the game lets you experiment with setting either of them as "true" and seeing the implications of this on your belief network. Or maybe they're both true - Bob is generally unreliable, and Alice is out to discredit him. That's another possibility that you might want to consider. In any case, the claim that there will be an assassination tomorrow isn't looking very likely at the moment.

Actually, having the possibility for somebody lying should probably be a pretty late-game thing, as it makes your belief network a lot more complicated, and I'm not sure whether this thing should display numerical probabilities at all. Instead of having to juggle the hypotheses of "Alice lied" and "Bob exaggerates things", the game should probably just record the fact that "Bob exaggerates things". But I spent a bunch of time making these pictures, and they do illustrate some of the general principles involved, so I'll just use them for now.

Game basics

So, to repeat the basic premise of the game, in slightly more words this time around: your task is to figure out something, and in order to do so, you need to collect different pieces of evidence. As you do so, the game generates a belief network showing the origin and history of the various pieces of evidence that you've gathered. That much is done automatically. But often, the evidence that you've gathered is compatible with many different hypotheses. In those situations, you can experiment with different ways of various hypotheses being true or false, and the game will automatically propagate the consequences of that hypothetical through your belief network, helping you decide what angle you should explore next.

Of course, people don't always remember the source of their knowledge, or they might just appeal to personal experiences. Or they might lie about the sources, though that will only happen at the more advanced levels.

As you proceed in the game, you will also be given access to more advanced tools that you can use for making hypothetical manipulations to the belief network. For example, it may happen that many different characters say that armies of vampire bats tend to move about at full moon. Since you hear that information from many different sources, it seems reliable. But then you find out that they all heard it from a nature documentary on TV that aired a few weeks back. This is reflected in your belief graph, as the game modifies it to show that all of those supposedly independent sources can actually be tracked back to a single one. That considerably reduces the reliability of the information.

But maybe you were already suspecting that the sources might not be independent? In that case, it would have been nice if the belief graph interface would let you postulate this beforehand, and see how big of an effect it would make on the plausibility of the different hypotheses if they were in fact reliant on each other. Once your character learns the right skills, it becomes possible to also add new hypothetical connections to the belief graph, and see how this would influence your beliefs. That will further help you decide what possibilities to explore and verify.

Because you can't explore every possible eventuality. There's a time limit: after a certain amount of moves, a bomb will go off, the aliens will invade, or whatever.

The various characters are also more nuanced than just "reliable" or "not reliable". As you collect information about the various characters, you'll figure out their mindware, motivations, and biases. Somebody might be really reliable most of the time, but have strong biases when it comes to politics, for example. Others are out to defame others, or invent fancy details to all the stories. If you talk to somebody you don't have any knowledge about yet, you can set a prior on the extent that you rely on their information, based on your experiences with other people.

You also have another source of evidence: your own intuitions and experience. As you get into various situations, a source of evidence that's labeled simply "your brain" will provide various gut feelings and impressions about things. The claim that Alice presented doesn't seem to make sense. Bob feels reliable. You could persuade Carol to help you if you just said this one thing. But in what situations, and for what things, can you rely on your own brain? What are your own biases and problems? If you have a strong sense of having heard something at some point, but can't remember where it was, are you any more reliable than anyone else who can't remember the source of their information? You'll need to figure all of that out.

As the game progresses to higher levels, your own efforts will prove insufficient for analyzing all the necessary information. You'll have to recruit a group of reliable allies, who you can trust to analyze some of the information on their own and report the results to you accurately. Of course, in order to make better decisions, they'll need you to tell them your conclusions as well. Be sure not to report as true things that you aren't really sure about, or they will end up drawing the wrong conclusions and focusing on the wrong possibilities. But you do need to condense your report somewhat: you can't just communicate your entire belief network to them.

Hopefully, all of this should lead to player learning on a gut level things like:

• Consider the origin of your knowledge: Obvious.
• Visualizing degrees of uncertainty: In addition to giving you a numerical estimate about the probability of something, the game also color-codes the various probabilities and shows the amount of probability mass associated with your various beliefs.
• Considering whether different sources really are independent: Some sources which seem independent won't actually be that, and some which seem dependent on each other won't be.
• Value of information: Given all the evidence you have so far, if you found out X, exactly how much would it change your currently existing beliefs? You can test this and find out, and then decide whether it's worth finding out.
• Seek disconfirmation: A lot of things that seem true really aren't, and acting on flawed information can cost you.
• Prefer simpler theories: Complex, detailed hypotheses are more likely to be wrong in this game as well.
• Common biases: Ideally, the list of biases that various characters have is derived from existing psychological research on the topic. Some biases are really common, others are more rare.
• Epistemic hygiene: Pass off wrong information to your allies, and it'll cost you.
• Seek to update your beliefs: The game will automatically update your belief network... to some extent. But it's still possible for you to assign mutually exclusive events probabilities that sum to more than 1, or otherwise have conflicting or incoherent beliefs. The game will mark these with a warning sign, and it's up to you to decide whether this particular inconsistency needs to be resolved or not.
• Etc etc.
  

Design considerations

It's not enough for the game to be educational: if somebody downloads the game because it teaches rationality skills, that's great, but we want people to also play it because it's fun. Some principles that help ensure that, as well as its general utility as an educational aid, include:

• Provide both short- and medium-term feedback: Ideally, there should be plenty of hints for how to find out the truth about something by investigating just one more thing: then the player can find out whether your guess was correct. It's no fun if the player has to work through fifty decisions before finding out whether they made the right move: they should get constant immediate feedback. At the same time, the player's decisions should be building up to a larger goal, with uncertainty about the overall goal keeping them interested.
• Don't overwhelm the player: In a game like this, it would be easy to throw a million contradictory pieces of evidence at the player, forcing them to go through countless of sources of evidence and possible interactions and have no clue of what they should be doing. But the game should be manageable. Even if it looks like there is a huge messy network of countless pieces of contradictory evidence, it should be possible to find the connections which reveal the network to be relatively simple after all. (This is not strictly realistic, but necessary for making the game playable.)
• Introduce new gameplay concepts gradually: Closely related to the previous item. Don't start out with making the player deal with every single gameplay concept at once. Instead, start them out in a trusted and safe environment where everyone is basically reliable, and then begin gradually introducing new things that they need to take into account.
• No tedium: A game is a series of interesting decisions. The game should never force the player to do anything uninteresting or tedious. Did Alice tell Bob something? No need to write that down, the game keeps automatic track of it. From the evidence that has been gathered so far, is it completely obvious what hypothesis is going to be right? Let the player mark that as something that will be taken for granted and move on.
• No glued-on tasks: A sign of a bad educational game is that the educational component is glued on to the game (or vice versa). Answer this exam question correctly, and you'll get to play a fun action level! There should be none of that - the educational component should be an indistinguishable part of the game play.
• Achievement, not fake achievement: Related to the previous point. It would be easy to make a game that wore the attire of rationality, and which used concepts like "probability theory", and then when your character leveled up he would get better probability attacks or whatever. And you'd feel great about your character learning cool stuff, while you yourself learned nothing. The game must genuinely require the player to actually learn new skills in order to get further.
• Emotionally compelling: The game should not be just an abstract intellectual exercise, but have an emotionally compelling story as well. Your choices should feel like they matter, and characters should be in risk of dying if you make the wrong decisions.
• Teach true things: Hopefully, the players should take the things that they've learned from the game and apply them to their daily lives. That means that we have a responsibility not to teach them things which aren't actually true.
• Replayable: Practice makes perfect. At least part of the game world needs to be randomly generated, so that the game can be replayed without a risk of it becoming boring because the player has memorized the whole belief network.

What next?

What you've just read is a very high-level design, and a quite incomplete one at that: I've spoken on the need to have "an emotionally compelling story", but said nothing about the story or the setting. This should probably be something like a spy or detective story, because that's thematically appropriate for a game which is about managing information; and it might be best to have it in a fantasy setting, so that you can question the widely-accepted truths of that setting without needing to get on anyone's toes by questioning widely-accepted truths of our society.

But there's still a lot of work that remains to be done with regard to things like what exactly does the belief network look like, what kinds of evidence can there be, how does one make all of this actually be fun, and so on. I mentioned the need to have both short- and medium-term feedback, but I'm not sure of how that could be achieved, or whether this design lets you achieve it at all. And I don't even know whether the game should show explicit probabilities.

And having a design isn't enough: the whole thing needs to be implemented as well, preferably while it's still being designed in order to take advantage of agile development techniques. Make a prototype, find some unsuspecting testers, spring it on them, revise. And then there are the graphics and music, things for which I have no competence for working on.

I'll probably be working on this in my spare time - I've been playing with the idea of going to the field of educational games at some point, and want the design and programming experience. If anyone feels like they could and would want to contribute to the project, let me know.

EDIT: Great to see that there's interest! I've created a mailing list for discussing the game. It's probably easiest to have the initial discussion here, and then shift the discussion to the list.