But suppose I give people a pre-test, have them play for some minimum time, and then fill out the post-test when they're done. I don't see much in the way for random chance to confound things here: either they know the things needed for solving the tasks, or they don't. If they didn't know enough to solve the problems on the first try, they're not going to suddenly acquire that knowledge in between.
If I get a problem I can't solve I can Google afterwards and read about how to solve the problem. Even if you lock me in a dark room, there the possibility that I recover forgotten knowledge if you give my brain a few hours.
The pretest itself also provides practice. You need a control group, but it would be possible to give the control group nothing to do.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.