Well, if we want to describe the world, the symmetry is broken by the fact that we can observe the world - our unique "language" is our observations of the natural world - which color is simpler when describing the neurons in our visual cortex, if you will.

That is the fundamental point, I agree. The overall description of our world includes our description of ourselves. It is here that the simplicity advantage of blue / green resides.