An ideal perceptual representation of sound would only need 2 hair cells - if hair cells, like cones, reported a distance from the stimulus.

That's true if you're talking about a stimulus that only contains a single frequency at a time, but real sounds and colors are mixtures of an entire spectrum of frequencies, each frequency having its own distinct amplitude.

For example, 2 hair cells, even if they had a wider frequency response, would not be enough to understand speech; for that you need at least 4 to 8 frequency bands.