What’s interesting is that, when you have to decide whether or not to gamble your first coin, the probability is exactly the same in the two cases (p=0.45 of a $2 payout). However, the rational course of action is different. What’s up with that?

Why on earth should we expect that the long term expected value of all future consequences of a choice to be equal to the immediate payoffs? They are two different things. Learning is the most obvious example of when these can be expected to be different. In this case learning information and in other cases learning skills.