See also: Challenging the Difficult and Tips and Tricks for Answering Hard Questions.

From Michael Nielsen's Reinventing Discovery:

In January 2009, [mathematician Tim] Gowers decided to use his blog to run a very unusual social experiment. He picked out an important and difficult unsolved mathematical problem, a problem he said he’d “love to solve.” But instead of attacking the problem on his own, or with a few close colleagues, he decided to attack the problem completely in the open, using his blog to post ideas and partial progress. What’s more, he issued an open invitation asking other people to help out. Anyone could follow along and, if they had an idea, explain it in the comments section of the blog. Gowers hoped that many minds would be more powerful than one, that they would stimulate each other with different expertise and perspectives, and collectively make easy work of his hard mathematical problem. He dubbed the experiment the Polymath Project.

The Polymath Project got off to a slow start. Seven hours after Gowers opened up his blog for mathematical discussion, not a single person had commented. Then a mathematician named Jozsef Solymosi from the University of British Columbia posted a comment suggesting a variation on Gowers’s problem, a variation which was easier, but which Solymosi thought might throw light on the original problem. Fifteen minutes later, an Arizona high-school teacher named Jason Dyer chimed in with a thought of his own. And just three minutes after that, UCLA mathematician Terence Tao—like Gowers, a Fields medalist—added a comment. The comments erupted: over the next 37 days, 27 people wrote 800 mathematical comments, containing more than 170,000 words. Reading through the comments you see ideas proposed, refined, and discarded, all with incredible speed. You see top mathematicians making mistakes, going down wrong paths, getting their hands dirty following up the most mundane of details, relentlessly pursuing a solution. And through all the false starts and wrong turns, you see a gradual dawning of insight. Gowers described the Polymath process as being “to normal research as driving is to pushing a car.” Just 37 days after the project began Gowers announced that he was confident the polymaths had solved not just his original problem, but a harder problem that included the original as a special case.

This episode is a microcosm of how intellectual progress happens.

Humanity's intellectual history is not the story of a Few Great Men who had a burst of insight, cried "Eureka!" and jumped 10 paces ahead of everyone else. More often, an intellectual breakthrough is the story of dozens of people building on the ideas of others before them, making wrong turns, proposing and discarding ideas, combining insights from multiple subfields, slamming into brick walls and getting back up again. Very slowly, the space around the solution is crowded in by dozens of investigators until finally one of them hits the payload.

The problem you're trying to solve may look impossible. It may look like a wrong question, and you don't know what the right question to ask is. The problem may have stymied investigators for decades, or centuries.

If so, take heart: we've been in your situation many times before. Almost every problem we've ever solved was once phrased as a wrong question, and looked impossible. Remember the persistence required for science; what "thousands of disinterested moral lives of men lie buried in its mere foundations; what patience and postponement... are wrought into its very stones and mortar."

"Genius is 1 percent inspiration, 99 percent perspiration," said Thomas Edison, and he should've known: It took him hundreds of tweaks to get his incandescent light bulb to work well, and he was already building on the work of 22 earlier inventors of incandescent lights.

Pick any piece of progress you think of as a "sudden breakthrough," read a history book about just that one breakthrough, and you will find that the breakthrough was the result of messy progress like the Polymath Project, but slower: multiple investigators, wrong turns, ideas proposed and combined and discarded, the space around the final breakthrough slowly encroached upon from many angles.

Consider what I said earlier about the problem of Friendly AI:

I doubt the problem will be solved by getting smart people to sit in silence and think real hard about decision theory and metaethics. If the problem can be solved, it will be solved by dozens or hundreds of people hacking away at the tractable edges of Friendly AI subproblems, drawing novel connections, inching toward new insights, drawing from others' knowledge and intuitions, and doing lots of tedious, boring work.

...This isn't the only way to solve hard problems, but when problems are sufficiently hard, then hacking away at their edges may be just about all you can do. And as you do, you start to see where the problem is more and less tractable. Your intuitions about how to solve the problem become more and more informed by regular encounters with it from all angles. You learn things from one domain that end up helping in a different domain. And, inch by inch, you make progress.

So: Are you facing an impossible problem? Don't let that stop you, if the problem is important enough. Hack away at the edges. Look to similar problems in other fields for insight. Poke here and there and everywhere, and put extra pressure where the problem seems to give a little. Ask for help. Try different tools. Don't give up; keep hacking away at the edges.

One day you may hit the payload.