Honestly, the guys who never wrote any vision algorithm should just go and stop trying to think about the subject too hard, they aren't getting anywhere sane with the intuitions that are many orders of magnitude off. That goes for much of the cognition evolution out there. We know the evolution can produce 'complex' stuff, we got that hammer, we use it on every nail, even when the nail is in fact a giant screw which is way bigger than the hammer itself, which ain't going to even move if hit with the hammer - firstly, it is big and secondly it needs entirely different motion - yet one who can't see size of the screw can insist it would. Some adjustments for the near vs far connectivity, perhaps even somewhat specific layering, but that's all the evolution is going to give you in the relevant timeframe for brains bigger than peanuts. That's just the way things are, folks.

People with early brain damage have other parts of brain take over, and perform nearly as well, suggesting that the networks processing the visual input have only minor optimizations for visual task compared to the rest, i.e. precisely the near-vs-far connectivity tweaks, more/less neurons per cortical column, that kind of stuff which makes it somewhat more efficient at the task.

edit: here's the taster: mammals never evolved extra pair of limbs, extra eyes, or anything of that sort. But in the world of the evolutionary psychologists and evolutionary 'cognitive scientists', mammals should evolve an extra eye or extra pair of limbs, along with much more complex multi-step adaptations, every couple millions years. This is outright ridiculous. Just look at your legs, look how much slower is the fastest human runner than any comparable animal. You'll need to be running for another ten millions years before you are any good at running. And in that time, in which monkey's body can barely adapt to locomotion on a flat surface again, monkey's brain evolves some complex algorithms for grammar of the human languages? The hunter-gatherer specific functionality? You got to be kidding me.

All of that evolutionary explaining of complex phenomena via vague handwave and talk of how beneficial it would've been in ancestral environment, will be regarded as utter and complete pseudoscience within 20..50 years. It's not enough to show that something was beneficial. The extra eye on the back, too, would have been beneficial for great many animals. They are given free pass to use evolution as magic because brains don't fossilize. The things that fossilize, however, provide good sample of how many generations it takes for evolution to make something.

I'm not expecting the specialization to be going beyond adjusting e.g. number of local vs long range connections, basic motion detection back from couple hundred millions years of evolution as a fish etc. People are expecting some miracles from evolution, along the lines of hard-coding specific highly nontrivial algorithms. Mostly due to very screwy intuitions regarding how complex anything resembling an algorithm is (compared to rest of the body). Yes, it is the case that other parts of brain are worse at performing this function, no, it isn't because there's some actual algorithms hard-wired there. In so much as other parts of brain are able to perform even remotely close in function, sight is learn-able. One particular thing about the from-scratch AI crowd is that it doesn't like to give much credit to brain when its due.