This discussion reminds me of something I read a few years ago. The question related to what makes a superior organization. Especially in a competitive environment, what are the success factors that deliver competitive advantage?

The answer, at a very high level, was that specific systems, specific initiatives, specific ideas are NOT the be-all and end-all. Any specific business project can deliver benefits, it's true, but those projects can be replicated (and often improved upon) by competitors. Your competitors you see, can learn from your mistakes. They also have the opportunity to set the bar higher, and create goals that are deeper and broader than whatever your original goals were.

No, the thing that durably successful organizations had going for them was their ability to come up with new ideas in the first place. They were idea generators, innovation machines. If you can build a learning culture, and a place where improvement is looked upon as a never ending process, and risks are embraced (controlled, but embraced), then you have a platform for enduring success.

Such an organization is ready for the future and can quickly adapt to present circumstances. Change is not seen as a problem, but as a source of opportunity.

The important issue here is what you do with this metrics. If you don't use it at all for improvement don't bother trying to measure. But, if you want to deploy a continual improvement cycle, you'll discover how high are metrics pay back.

Diagnosticated brings an interesting and fresh viewpoint, although I think he or she is really just trying to stir up trouble by questioning this accepted wisdom that you cannot manage what you cannot measure.

But seriously, I certainly think there are times when there are no adequate metrics to determine success. When we implemented our CRM a few years ago, the vendor thought we should somehow relate the success of our business to the implementation of the CRM system.

I asked whether he meant that, after a year of the new system, we could give credit to the CRM if our sales or revenues went up. He agreed until I asked if the converse would hold true - that we could blame them if revenues went down.

I finally told him that, because I saw no useful metrics that could measure cause and effect, *ANY* time and effort spent trying to measure the ROI would actually reduce the ROI of the CRM investment.

Sometimes it comes down to quality versus quantity - we have the term "qualitative" because some things are subjective and difficult to measure. This is not to say that people cannot intuitively tell whether something is good or bad, or better or worse.

This is not the first time I've pointed people towards Robert Austin's book MEASURING AND MANAGING PERFORMANCE IN ORGANIZATIONS (Dorset House, 1996). Austin is now a professor at HBS, but at the time he was an executive with the Ford Motor Co. Europe. In this quite readable book, he applies a version of Game Theory, called Agency Theory, to show that in an information economy, metrics can be easily "gamed". There are a lot of popular stories about call center agents hanging up on customers, PC support people sending entirely new PCs to customers who had minor problems, etc., all in the name of getting better metrics. Austin argues that you can't measure (and maybe can't even define) all the dimensions of metrics that really indicate whether or not someone is doing a good job; people will game the numbers you aren't measuring in order to improve the ones you are. Pretty compelling stuff.

The start of this question is what is it that you want to consider changing?

If you really don't have anything under consideration for change then frankly measuring anything has no return available regardless of the investment. This may be almost as important as the concept that "you tend to get what you measure". If you are considering changing something then the return on investment is the amount that could be invested in the change. Even if you didn't make the change because your metric showed it to not be a good investment then you had a return on your investment in your metric. This is probably the most valuable return on this investment when no money actually changes.

I think the SMART idea is useful when folks are not performing up to expectations -- in which case their gaming of the system may be acceptable, as long as they are finally getting the work done. But that is also (necessarily?) a form of micromanaging -- and I doubt that a CIO could effectively micromanage an entire IT organization.

Since the "S" stands for "Specific", doesn't it also mean you never do anything novel? (how can you be specific about a potential game-changing project or idea that you've never done before?)

Metrics, almost as useful as mission statements.

How about this for a metric:

How long between the time a customer dials the phone and that person gets a good fix to whatever problem they are experiencing?

As Bob constantly reminds us, optimizing the part often de-optimizes the whole. But wouldn't it be great to talk to the person who can actually fix a problem when you call the company? And I really don't care if I have a little difficulty with the accent. Sometimes you get better service by selecting the Spanish option on the menu.

John P raises an interesting point. My spousal unit (Mrs. Overclock, a.k.a. Dr. Overclock, Medicine Woman) and I spent three weeks traveling around Japan last summer. The culture shock caused me to think a lot about the Culture of Innovation (Western) versus the Culture of Efficient Production (Eastern). I'm beginning to come around to the opinion that the two are in direct conflict -- which is why many organizations spin off their research arm into a Bell Labs or Xerox PARC. It is unreasonable to expect innovation in a corporate culture where the emphasis is on short term sharholder value, efficient mass production, lowering costs, etc. Innovation comes from making a lot of mistakes (according to a two-time Nobel winner), being willing to try out radically new ideas, and just futzing around. Having spent the past decade-plus in product development in several large corporations, I just don't see where there's room for that anymore in the schedule or the budget. Which I think will eventually lead me to conclude that innovation or creativity will generally not occur in the typical large development organization. I'm not quite there yet, but that's the light at the end of the tunnel I think.

Innovation comes from making a lot of mistakes (according to a two-time Nobel winner)

Who?

"The best way to have good ideas is to have lots of ideas." -- Linus Pauling

May be be apocryphal, but widely quoted, frequently by me. But supported by a lot of other related concepts like "fail often, fail early" etc.

Gostak - The only true measurement of success for a call center should be customer satisfaction. Call throughput or durations or tickets flagged as "issue resolved" are so thoroughly open to gaming the system as to be not useful for anything.

Indeed, the best call center sees the number of calls go down as issues are resolved, callers are trained or otherwise guided so that the problem does not recur or recurs less often.

This also means that call center staff need to be empowered to actually solve problems.