You need some external way of verifying the results, and you need to be careful that you are still interpreting the external data correctly and didn't just self-modify yourself to go insane. (You can test yourself on data you've generated yourself, and where you know the correct answers, but that doesn't yet show that you'll process real-world data correctly.)

If I was an AI in such a situation, I'd make a modified copy of myself (or of the relevant modules) interfaced with a simulation environment with some physics-based puzzle to solve, such that it only gets a video feed and only has some simple controls (say, have it play Portal - the exact challenge is a bit irrelevant, just something that requires general intelligence). A modified AI that performs better (learns faster, comes up with better solutions) in a wide variety of simulated environments will probably also work better in the real world.

Even if the combinations of parameters that makes functional intelligence is very fragile, i.e. the search space has high-dimensionality and the "surface" is very jagged, it's still a search space that can be explored and mapped.

That's a bit hand-wavy, but enough to get me to suspect that an agent that can self-modify and run simulations of itself has a non-negligible chance of self-improving successfully (for a broad meaning of "successfully", that includes accidentally rewriting the utility function, as long as the resulting system is more powerful).

But the five year estimate doesn't strike me as unreasonable.

Meaning, a 1% chance of superhuman intelligence within 5 years, right?