Apparently HICANN was designed before 2008, and uses a 180nm CMOS process, whereas modern GPUs are using 28nm.

That's true, but IBM's TrueNorth is 28 nm, with about the same transistor count as a GPU. It descends from earlier research chips on old nodes that were then scaled up to new nodes. TrueNorth can fit 256 million low-bit synapses on a chip, vs 1 million for HICANN (normalized for chip area). The 28 nm process has roughly 40x the transistor density. So my default hypothesis is that if HICANN was scaled up to 28 nm it would end up similar to TrueNorth in terms of density (although TrueNorth is wierd in that it is intentionally much slower than it could be to save energy).

It seems to me that if neuromorphic hardware catches up in terms of economy of scale and process technology, it should be far superior in cost per neural event.

I expect this in the long term, but it will depend on how the end of Moore's Law pans out. Also, current GPU code is not yet at the limits of software simulation efficiency for ANNs, and GPU hardware is still improving rapidly. It just so happens that I am working on a new type of ANN sim engine that is 10x or more faster than current SOTA for networks of interest. My approach could eventually be hardware accelerated. There are some companies already pursuing hardware acceleration of the standard algorithms - such as Nervana, targeting similar speedup but through dedicated neural asics.

One thing I can't stress enough is the advantage of programmeable memory for storing weights - sharing and compressing weights helps solve much of the bandwidth problems the GPU would otherwise have.

It seems like this GPU vs neuromorphic question could have a large impact on how the Singularity turns out, but I haven't seen any discussion of it until now. Do you have any other thoughts or references on this topic?

I don't know much it really effects outcomes - whether one uses clever hardware or clever software, the brain is probably near or on the pareto surface for statistical inference energy efficiency, and we will probably get close in the near future.