Another quote from the same piece, just before that para: [...] I really, really like this. Thanks for posting it! To elucidate the "bug model" a bit, consider "bugs" not in a single piece of software, but in a system. The following is drawn from my professional experience as a sysadmin for large-scale web applications, but I've tried to make it clear: Suppose that you have a web server; or better yet, a cluster of servers. It's providing some application to users — maybe a wiki, a forum, or a game. Most of the time when a query comes in from a user's browser, the server gives a good response. However, sometimes it gives a bad response — maybe it's unusually slow, or it times out, or it gives an error or an incomplete page instead of what the user was looking for. It turns out that if you want to fix these sorts of problems, considering them merely to be "flakiness" and stopping there is not enough. You have to actually find out where the errors are coming from. "Flaky web server" is an aggregate property, not a simple one; specifically, it is the sum of all the different sources of error, slowness, and other badness — the disk contention; the database queries against un-indexed tables; the slowly failing NIC; the excess load from the web spider that's copying the main page ten times a second looking for updates; the design choice of retrying failed transactions repeatedly, thus causing overload to make itself worse. There is some fact of the matter about which error sources are causing more failures than others, too. If 1% of failed queries are caused by a failing NIC, but 90% are caused by transactions timing out due to slow database queries to an overloaded MySQL instance, then swapping the NIC out is not going to help much. And two flaky websites may be flaky for completely unrelated reasons. Talking about how flaky or reliable a web server is lets you compare two web servers side-by-side and decide which one is preferable. But by itself it doesn't let yo

Great article. One thing: [...] I don't know much about Knewton, but it seems like it could address this - at least in some cases - and possibly better than teachers. Knewton and programs like it can keep track of success rates at the individual problem level, rather than the test or semester level. Such data could be used to identify the 'bugs' the author speaks of. All Knewton needs is knowledge of common 'bugs' and what problems they make students get wrong. This article also recalls to mind http://lesswrong.com/lw/6ww/when_programs_have_to_work_lessons_from_nasa/, specifically the part where problems are considered to be the fault of the system, not of the people involved and are treated by changing the system, not by criticizing the people.