I started writing a reply to this comment, but as I was thinking through it I realized that the situation is actually WAY more interesting than I thought and requires a whole post. I've posted it in discussion:

http://lesswrong.com/r/discussion/lw/is7/the_cause_of_time/

Sorry if it's a bit unclear right now, hopefully I'll have time to add some diagrams this weekend.