Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

pjeby comments on The Importance of Goodhart's Law - Less Wrong

75 Post author: blogospheroid 13 March 2010 08:19AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (113)

You are viewing a single comment's thread.

Comment author: pjeby 14 March 2010 07:32:02PM *  27 points [-]

There are no generic solutions to bridging the gap between G and G*, but the body of knowledge of theory of constraints is a very good starting point for formulating better measures for corporates.

A good example from my own history of doing this is when I worked for an ISP and persuaded them to eliminate "cases closed" as a performance measurement for customer service and tech support people, because it was causing email-based cases to be closed without any actual investigation. People would email back and create a new case, and then a rep would get credit for closing that one without investigation either.

The replacement metric was one I derived via the Theory of Constraints, inspired by Goldratt's "throughput-dollar-days" measurement. The replacement metric was "customer-satisfaction-waiting-hours" - a measurement of collective work-in-progress inventory at the team level, and a measurement of priority at the ticket level.

I also made it impossible to truly "close" a case - you could say, "I think this is done", but the customer could still email into it and it would jump right back to its old place in the queue, due to the accumulated "satisfaction waiting hours" on the ticket.

Of course, the toughest part in some ways was educating new service managers that, no, you can't have a measurement of cases closed on a per-rep basis. Instead, you're going to have to actually pay attention to a rep's work in order to know if they're doing the job. (Of course, the system I developed also had ways to make it easy to see what people are working on, not only at the managerial but the team level - peer pressure is a useful co-ordination tool, if done right.)

I have no idea how well the system fared since I left the company, since it's entirely possible they found programmers since then to give them new metrics that would f**k it up, although I did design the database in such a way as to make it as close to impossible as I could manage. ;-)

Anyway, the theory of constraints positively rocks for business performance optimization, and its Thinking Processes are generally useful tools for any rationalist. They were also a big inspiration for me developing other thinking processes and ultimately mindhacking techniques, in that they showed that it's possible to think systematically even about some of the vaguest and most ill-defined problems imaginable, rigorously hone in on key leverage points, resolve conflicts between goals, and generally overcome our brains' processing limitations for analysis and planning.

[Edit to add: the Wikipedia page on thinking processes doesn't really show why a rationalist would be interested in the processes; it's useful to know that a key element of the processes are something called the "categories of legitimate reservation", which have to do with logical proof and well-formedness of argument. They are a key part of constructing and critiquing the semantic maps that are created by the thinking processes.

For example, ToC's conflict resolution method effectively maps out certain implicit assumptions in a conflict, and then invites you to logically disprove these assumptions in order to break the conflict. (That is, if you can find a circumstance where one of those assumptions is false, then the conflict will no longer exist under that circumstance - and you have a potential way out of your dilemma.)

So, in short, ToC thinking processes are mostly about constructing past, present, or future semantic maps of a situation, and applying systematic logic to validating (or invalidating) the maps' well-formedness, as a way of solving problems, creating plans, etc. Very core rationalist stuff, from an instrumental-rationality POV.]