"Prove would work" is a much more stringent standard than "just be smarter and have the same goals." The import of the paper is that the budding FOOMster couldn't prove that a change would work.

Lacking a proof, if the imperfection lies in the modeling of intelligence improvement, it could be in error when believing the update will make it smarter and have the same goals.

I believe Nancy's point is a very good one. Intelligence has generally improved in evolutionary fashion. Some things work, some don't. People seem to picture the AI as a fully self referential optimizer, which per Wolpert would be a mistake, and more generally a mistake as I think fully recursive self reference explodes without bounds in complexity as you iterate your self referencing.

Instead, you have some simple rules applied to some subset self referentially, which seems to improve some local issue, but may fail in a wider context. In the end, you make a guess based on your current functioning and reality tells you if you were right.