Seems to me that you'd want to add up the probabilities of each of the 10 outcomes, 0*p^10*(10!/(10!*0!)) + 9000*p^9*(1-p)*(10!/(9!*1!)) + 8000*p^8*(1-p)^2*(10!/(8!*2!)) + 7000*p^7*(1-p)^3*(10!/(7!*3!))... This also has a maximum at p~= 0.774, with expected value of $6968. This verifies that your shortcut was correct.

James' equation gives a bigger value, because he doesn't account for the fact that the lost payoff is always the maximum $10,000. His equation would be the correct one to use, if the problem were with 20 people, 10 of which determine the payoff and the other 10 whether the payoff is payed and they all have to use the same probability.