Camus may answer along the lines of "since [any ascribing of meaning] is absurd in the first place, if you think there's objectively more meaning in the building you built than in the rock you pushed up, you're not taking the premise seriously". In a way we're whistling in a dark forest.