I think money might complicate things: You might want to get paid more for stuff you don't find that interesting. With trading just time, it feels different. You'd just give the other person X hours of your time, and you get X hours back. It doesn't matter to you what you do in the X hours you gave away. Perhaps getting money for it also makes it seem like work, instead of a fun, social thing. Then again, maybe it's a distinction that's only in my head, so if you can make it work, sure, go for it!

Buying food indeed seems less formal.