Neither probability should be <50%, you take the probability that your opinion is the right one, not whether the proposition is true or false.

In your example B would be betting against his beliefs, thus the negative result.

The right calculation: A = 0.6 B = 0.7

A pays: (A ^ 2 - (1 - B) ^ 2) * 25 = (0.36 - 0.09) * 25 = 6.57
B pays: (B ^ 2 - (1 - A) ^ 2) * 25 = (0.49 - 0.16) * 25 = 8.25

Edit:

actually, it's sufficient that A and B sum to over 1. Since you can always negate the condition, the right calculation here is:

A = 0.4
B = 0.7
A pays: (A ^ 2 - (1 - B) ^ 2) * 25 = (0.16 - 0.09) * 25 = 1.75
B pays: (B ^ 2 - (1 - A) ^ 2) * 25 = (0.49 - 0.36) * 25 = 3.25

Also, apparently I can't use the retract button the way I wanted to use it.