Yes, this is one of the issues that does need to be thought about. But there are limits to this. However, there's been work in the last few years (especially for problems in graph theory) where one can show that being able to find close to optimal solutions is equivalent to being able to find optimal solutions.

The encryption example is also an interesting one, in that many foom scenarios involve the hypothetical AI gaining control over lots of the world's computer systems. And in fact, no known encryption system rest on a problems which is NP-complete, so for all we know we could have P !=NP in a strong sense and yet still have essentially all encryption be vulnerable. Also, the AI might find security vulnerabilities completely independent of the math due to implementation issues.

Another reason that P != NP might not be a severe barrier that cousin_it pointed out is that our AI would be unlikely to need to solve general NP-complete problems, just real-world instances which are likely to have additional regularity in them. Moreover, for most NP-complete problems, random instances of the problems are generally easy (indeed, this is part of what went wrong in Deolalikar's attempted proof. The general behavior on aggregate of NP-complete problems like K-sat for fixed k>=3 looks a lot like 2-SAT which is in P). So something trying to foom might not even run into such instances when trying to engage in practical applications of NP hard problems (such as where say graph coloring comes in optimizing memory allocation.) And there are some NP hard problems where one really does care about some restricted form of the problem. One example of this is protein folding where it seems that most models of protein folding are at least NP hard.

However, when trying to foom, an AI will probably want to directly improve the algorithms it is using on a regular basis. While not directly a P/NP issue, it is noteworthy that for some common procedures, such as finding gcds, and solving linear programming, we have what are in many ways close to optimal algorithms (the situation is more complicated for linear programing than that but the general point is correct). So the AI probably cannot focus primarily on modifying software because a lot of the basic things that the AI would want to optimize are already close to best possible even for both the average cases and the worst cases.

There are also other problems with this sort of thing. So for example, it might turn out that P != NP but that there's a general NP-complete solver that runs in almost polynomial time with a really small constant in the Big-Oh. There's as I understand it some very weak results showing that this can't happen for some functions very close to polynomial growth.

There are other things that could go wrong. If for example, quantum computers are much more powerful than classical systems then the AI could if it had access to one just side-step almost all of these issues. We can't for example even show that BQP is contained in NP. If BQP is larger than anticipated then this is a serious problem. Moreover, even if P !=NP in a strong sense, it is plausible that there's a quantum algorithm which can solve NP-complete problems in practically close to polynomial time (this is more plausible than the similar idea in the previous paragraph about the classical case). So, if we end up with practical quantum computing, running an AI on it would be a bad idea.

There are also a handful of more exotic possibilities. Scott Aaaronson has discussed using minature wormholes to aid computation. And if you can do this, then all of PSPACE collapses. From the perspective of an AI trying to foom, this is good. From our perspective, probably not. The good news here is that the tech level required is probably well beyond anything we have and so no AI would probably have access to this sort of thing unless it had already foomed or was so close to fooming that it makes no difference.

The overall upshot of this is that while it is possible that an AI with access to a quantum computer could plausibly foom with most of the focus on software improvement, comp sci issues suggest that an AI trying to foom on a classical system would need to engage in both hardware and software improvement. Trying to improve your hardware is tough without either prior access to molecular nanotech(which is its own can of nastiness regardless of whether humans or AI have access) or a lot of cooperative humans who have given one access to circuit board and chip manufacturing facilities. Moreover, hardware generally produces diminishing marginal returns as long as the hardware remains the same type (i.e. just improving on classical computers) and could run into its own problems as design issues themselves involve problems which are computationally complex (although cousin_it again has pointed out that this could be less of a problem than one might think since there are regularities that might show up).

I will be able to sleep better at night once we've proven that P != NP.

(Ok. There's a lot of material here, and some of it probably should get expanded. How would people feel about expanding this into a top-level post with nice citations, better organization, and all that?)