For example, if you try to make very small, spherical diamond crystals, a layer or two of carbon atoms at the surface will spontaneously rearrange themselves into a new form--not of diamond, but of graphite.

What do we count as "spherical"? Adamantane is a symmetrical 10 carbon atom piece of a diamond lattice, with surface bonds terminated with hydrogen atoms. It is stable enough to be melted at 270C and recrystallized from the melt. It does not rearrange itself into graphite.

More generally: AFMs and STMs routinely use atomically precise positioning in the presence of thermal noise (the vibrational analog of Brownian motion) at room temperature. Set aside Drexler's analyses of thermal and quantum motions in molecular scale device for a moment: At this point we've had multiple decades of experimental experience of atomically precise positioning at room temperature. The tips of these devices are molecular scale structures being positioned with atomic precision. Sufficient?