Open Thread, Apr. 27 - May 3, 2015

Gondolinian

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should be posted in Discussion, and not Main.

4. Open Threads should start on Monday, and end on Sunday.

Additionally, I'm a little worried about the control group part. I expect it's relatively easy to recruit people to play a game and have them be motivated to play it, but if I tell people that "oh, but you may be randomly assigned to the control condition where you're given more traditional math instruction instead", I expect that that will drop participation. And even the people who do show up regardless may not be particularly motivated to actually work on the problems if they do get assigned to the control condition, especially given that I'm hoping to also educate people who'd usually avoid maths. How insane would it be to just not have a control group?

If you didn't have any control group, you wouldn't be able to interpret any improvement between pretest and posttest, if you observed such a pattern: repetition or practice effects could explain any improvement. If you observed no improvement, you wouldn't need a control group because there's no effect to be explained.

Sometimes exploratory methods start out with no-control group pilots just to see if a method is potentially promising (if no hints of effects, don't invest a lot of resources in trying to set up a proper study).

Sometimes studies like this ar... (read more)

6afeller0811y

If I were designing the experiment, I would have the control group be to play a different game instead of having it be maths instructions. You generally don't want test subjects to know whether they are in the control condition or not. So if you're going to make it be maths instructions, you probably shouldn't tell them what the experiment is designed to test at all, until you're debriefing at the end. If you tell people you are recruiting that you are testing the effects of playing computer games on statistical reasoning, then the people in the control condition won't need to realize that what you're really testing is whether your RPG in particular helps people think about statistics. They can just play HalfLife 2 or whatever you pick for them to play for a few minutes, and then take your tests afterwards.

7TylerJay11y

Pretty insane in my opinion. I can't imagine anything I would grade more harshly than not having a control except ethics violations. Besides, don't most university psychology experiments with volunteers keep the protocol secret throughout the whole experiment and then debrief at the end? (Or sometimes even lie about the protocol to avoid skewing the results?) Alternatively, have you thought about doing a crossover-style design? Take group A and group B. Group A plays your game, and then takes the test. Group B either just takes the test or goes through some traditional education lesson (or whatever else you want for your control) and then takes the test. Next, group A does the traditional education, group B does the game, and both take part 2 of the test. That way, everyone gets to play the game at least, though it means they're there for twice as long. Do you think you could pitch this in a way that is better than the "Maybe you play a game, maybe you don't" option? You could potentially derive additional research value from this as well. If group A does better on Test Part 2, then your game would be shown to be a better way of acclimating people to traditional education on the subject or something like that (I'm sure you can draw a better conclusion or phrase this better). Just some thoughts. Also, make sure you write up a grading rubric ahead of time (or ideally, have someone else do it) and then have someone who knows nothing (or as little as possible) about the experiment (and especially the subjects) grade the answers to avoid researcher bias.