Is it necessary that we understand how intelligence works for us to know how to build it? This may almost be a philosophical question. Certainly a guy who builds race car engines almost certainly knows nothing about the periodic table of elements and the quantum effects behind electronic orbitals that can explain some of the mechanical properties of the metals that are used in the engines. Very likely he does not know much thermodynamics, does not appreciate the interplay between energy and entropy required to make a heat engine produce mechanical power. Possibly knows very little of the chemistry behind the design of the lubricants or the chemistry evolved in storing energy in hydrocarbons and releasing it by oxidizing it.
But I'd sure rather drive a car with an engine he designed in it than a car with an engine designed by a room full of chemists and physicists.
My point being, we may well develop a set of black boxes that can be linked together to produce AI systems for various tasks. Quite a lot will be known by the builders of these AIs about how to put these together and what to expect in certain configurations. But they may not know much about how the eagle-eye-vision core works or how the alpha-chimp-emotional core works, just how the go together and a sense of what to expect as they get hooked up.
Maybe we never have much sense of what goes on inside some of those black boxes. Just as it is hard to picture what the universe looked like before the big bang or at the center of a black hole. Maybe not.
Is it necessary that we understand how intelligence works for us to know how to build it? This may almost be a philosophical question.
This is definitely an empirical question. I hope it will be settled "relatively soon" in the affirmative by brain emulation.
Claim: The first human-level AIs are not likely to undergo an intelligence explosion.
1) Brains have a ton of computational power: ~86 billion neurons and trillions of connections between them. Unless there's a "shortcut" to intelligence, we won't be able to efficiently simulate a brain for a long time. http://io9.com/this-computer-took-40-minutes-to-simulate-one-second-of-1043288954 describes one of the largest computers in the world simulating 1s of brain activity in 40m (i.e. this "AI" would think 2400 times slower than you or me). The first AIs are not likely to be fast thinkers.
2) Being able to read your own source code does not mean you can self-modify. You know that you're made of DNA. You can even get your own "source code" for a few thousand dollars. No humans have successfully self-modified into an intelligence explosion; the idea seems laughable.
3) Self-improvement is not like compound interest: if an AI comes up with an idea to modify it's source code to make it smarter, that doesn't automatically mean it will have a new idea tomorrow. In fact, as it picks off low-hanging fruit, new ideas will probably be harder and harder to think of. There's no guarantee that "how smart the AI is" will keep up with "how hard it is to think of ways to make the AI smarter"; to me, it seems very unlikely.