Okay, I've got it. You're a selflessly devoted altruist, and you know that money in your hands doesn't matter any more than money elsewhere - but with a $1 bill, you could generate gains from trade and thereby make the whole world a slightly better place!
Nice. Steven Landsburg once wrote something similar (I think in The Armchair Economist) about how it is wasteful to bend down to pick up a $50 bill off the sidewalk, because the money is a pure transfer and the effort of bending down is a pure social loss.
The problem is that I pretty clearly don't value a random person having the $4 as much as me having it, as evidenced by the fact that a lot of my income I keep, and the part of it that I give away (which is pretty substantial but the Peter Singer argument is always working on me that it should be higher) I give away to really poor people abroad, not to random Americans.
The other day I went to get some productivity-enhancement M&Ms from the candy machine at work. When I opened my wallet, I didn't immediately see a $1 bill. Then I looked some more and I found one, and I was happy! But of course that doesn't make any sense. If that bill hadn't been a $1, then it would have had to be a $5 or more, with an expected value of $5+, which is an amount that I certainly would not have paid for a bag of M&Ms, most excellent though they may be. This means that I preferred a bag of M&Ms to $1 (that's why I went to the candy machine in the first place), $1 to $5+ (I was happy when the bill turned out to be a $1), and $5+ to a bag of M&Ms (I wouldn't have bought them at that price). Not too surprising I guess, but still kind of weird.