As a pedantic note, if you want to derandomize algorithms it is necessary (and sufficient) to assume P/poly != E, i.e. polynomial size circuits cannot compute all functions computed by exponential time computations. This is much weaker than P != NP, and is consistent with e.g. P = PSPACE. You don't have to be able to fool an adversary, to fool yourself.

This is sometimes sloganized as "randomness never helps unless non-uniformity always helps," since it is obvious that P << E and generally believed that P/poly is about as strong as P for "uniform" problems. It would be a big shock if P/Poly was so much bigger than P.

But of course, in the worlds where you can't derandomize algorithms in the complexity-theoretic sense, you can still look up at the sky and use the whole universe to get your randomness. What this means is that you can exploit much of the stuff going on in the universe to do useful computation without lifting a finger, and since the universe is so much astronomically larger than the problems we care about, this is normally good enough. General derandomization is extremely interesting and important as a conceptual framework in complexity theory, but useless for actually computing things.