Yes, you're correct that in the majority of cases, the age of the building makes no difference (e.g. no one recognizes the difference between a two year old building and a twenty year old building) . . .

My dorm building has the number 2008 carved conspicuously into one of the stones in its facade. It's pretty easy to tell that it's a two year old building.