Understanding the OS to be able to optimize better sounds somewhat useful to a self-improving AI.
Understanding the OS to be able to reason properly about probabilities of hardware/software failure sounds very important to a self-improving AI that does reflection properly. (obviously it needs to understand hardware as well, but you can't understand all the steps between AI and hardware if you don't understand the OS)
If I remember correctly then a VHDL specification of hardware was also part of the STEPS project.
The View Point Reasearch Institute http://www.vpri.org/ (which by the way brought us the squeek etoys) has a project called STEPS Toward The Reinvention of Programming.
The goal is basically to create a complete operating system, tool chain and program suite in 20000 lines of code. The reason for this and other goals are not relevant for this post but I recommend that you take a look into it anyway.
I have followed the progress of the STEPS project more or less since its beginning in 2007: http://www.vpri.org/pdf/tr2007008_steps.pdf
This project is interesting to the (F)AI community in so far as it contains a complete operating system in a form that contains more or less no noise or redundancy in its implementation, is extremely powerful (in terms of expressivity and generality) and thus more amenable to artificial reasoning and optimization than any other system.
The current report http://www.vpri.org/pdf/tr2011004_steps11.pdf contains a very interesting paragraph on page 22:
"An awkward and complicated ʹoptimization tower of babelʹ is avoided by giving our DSLs the ability to act on their own implementation and optimizations, flattening the ʹtowerʹ into a collection of reflexive and metacircular mechanisms. The line between strategy and implementation, between coordination and computation, is eliminated ‑‑ or at least guaranteed to be eliminable whenever necessary."
That means that an AI system running on top of that could optimize itself as is needed for an intelligence explosion I guess. And you may guess who might be running AI on top of it.
EDIT: Some seem to think that this post is somewhat inappropriate. But I can't make out why. Because it is too short? Because it is about AI? Because it requires reading up on that project? Because it is about software engineering instead of rationality? I'm a newbie and don't know which or how to improve. Please comment your negative rating.