A standard Bayesian problem would work great. I paid my 13 year old nephew $1 to solve one.
Also: If you call a tail a leg, how many legs does a horse have?
Be careful how you reward people for mental tasks if you care about the long term cultivation of their mind.
I'm looking for hard philosophical questions to give to people to gauge their skill at philosophy.
So far, I've been presenting people with Newcomb's problem and the Sleeping Beauty problem. I've also been presenting them with contrarian opinions and asking them to evaluate them, and I have a higher opinion of them if they avoid just icking away from the subject.
What other problems should I use?