Thanks for sharing this! I’ve read Feedbackloop-first Rationality, and it’s definitely contributed why I want to build something like this. I’ve even been looking for Thinking Physics style problems that might be free to use online. Getting a diverse and quality set of interesting problems I think will be difficult whether its aggregated, crowdsourced, or possibly AI generated. 

My agenda is written up in the Feedbackloop-first Rationality sequence. The basic idea is that rationality is bottlenecked on inventing better feedbackloops that train the actually important skills. (You can look over the "exercises" section)

This will be very important because if a tool like this can be used to practice something hundreds of times. It should of course be something you actually want to reinforce.  If designed poorly, it could become a waste of time or, worse, actually harm one's thinking. I'll definitely take a look at more of these exercises.

• My general strategy has been to take existing puzzles/exercises that have a fair amount of depth, such that in order to solve it you're going to need to:
• make a plan for gaining more information about the puzzle
• make a plan for acting on that information

It might be good to have an answer template that guides users through these kinds of thinking steps. This could help build the habit of thinking systematically. With so many problem types, though, it’s challenging to settle on effective schemas that could apply to multiple problems.

I appreciate the reply! 

Something about training people to categorize errors - instead of just making good decisions - rubs me the wrong way

Are you able to pinpoint exactly what gives you this feeling? The goal of this problem type would be to train the ability to recognize bias to the point where it becomes second nature, with the hope that this same developed skill would also trigger in your own thought processes. I believe it’s generally easier to evaluate the truthfulness of a statement than to come up with one initially, so this training would help make the "biased thought detector" more accurate. 

Relatedly, the "ask users to predict an outcome based on limited data" example sounds like a description of that genre I invented (though "Bite-Sized" suggests you're thinking in terms of something much more polished/generally-accessible).

That’s really cool! I definitely see the value in multi-step case study problems, as they would require more complex reasoning than smaller bite-sized problems might. Themed problems could make the process much more engaging as I think this kind of training can get a bit dull with overly generic examples. Combining the depth of case studies with the accessibility of simpler exercises might strike a nice balance. 

I look forward to seeing what comes of this. If you want anything playtested, please let me know.

Definitely will take you up on this! I'm working on the prototype and should have something simple in the next few weeks. I'm considering starting a sequence to document the progress to get more visibility, interest, and immediate feedback.

Thanks! I have seen a similar tool like this before and enjoyed it quite a bit. I’d love to know where you source the trivia data, especially if it is available for open use. Also could be interesting to tailor to some functionality for meetups as well.