If you look in the appendix, you'll see we ran something similar to that, where we told them to "consider 10". It increased solution rates from 13% to 19%.
I partially agree. But I'll push back a little. The 23% wrong answers are not random key mashing. 80% of them type the number "10". Instead of copying the single digit that they are instructed to copy, they are looking further from the answer blank, finding two three-digit numbers, subtracting one from the other, and typing the two-digit difference. That is more work than the correct answer, not less.
But still, I do agree that if people had accuracy incentives, the correct answer rate would probably go up.... Then again, maybe, with monetary incentives, people would be even more likely to think the provided answer was a trap. If I had to bet, I'd say that incentives would increase the 77% "5" rate. But I wouldn't be totally shocked if it went down.
Monetary incentives raise solution rates a little, but not that much. Lex Borghans and co-authors manipulate small incentives and they do almost nothing. Ben Enke and co-authors offers a full month's salary to Nairobi based undergrads and finds a 13% percentage point increase in solution rates.
I'm not sure how our manipulations would interact with monetary incentives. But I'd like to know!