No matter how often I argue that metrics won"t be used improperly, the only way I"ve won over my clients is to show them the metrics. Let"s take accuracy in the form of rework. When designing the metric, if you stick to the rule that it has to be from the customers" viewpoint, you won"t even collect data on internal rework. You will only deal with rework from the customer"s point of view.
When I was designing effectiveness metrics for our printing services, the question arose around what const.i.tuted rework.
"If we have to send the printer back to the vendor three times before they fix it, is that rework?"
I answered with a question. "Does the customer know you sent it back three times?"
"No."
"Then no, it"s not rework."
"How about if we get the wrong parts from the vendor?"
Again, I replied with a question, "Does the customer know?"
"Nope."
"Then it"s still not rework."
"How about if we fix a printer jam, and when the customer tries to print, we find out there was also another problem?"
"Does the customer know?"
"Yes, but it wasn"t our fault..."
"Then, yes, that"s rework."
"But it wasn"t our fault..."
"I understand that. But the metric isn"t used to place blame. The metric is simply a way for us to understand what the customer sees. If the customer would consider it rework, we want to know. We can"t solve a problem if we don"t know it exists."
Effectiveness is not used to judge or place blame. Effectiveness metrics provide us the customers" view of our services and products.
Focusing on First Things First.
You have to start at the beginning, while keeping the end in mind. The end is a well-functioning, comprehensive metrics program. To get there though, we have to have a mature organization. We have to have a culture of trust. We must have open communications in all directions. We have to be pretty well ahead of the curve. As David Allen said in his book Getting Things Done (Penguin, 2002), about his consulting services, "those who use [his methods for improvement] need it the least."
So for the rest of us, we have to start at the beginning. And the first thing you"ll need to do is see your organization as your customer sees it. Effectiveness measures do that.
Recap.
Start at the beginning. It keeps you out of a lot of trouble when you start at the beginning. Jumping into mature behaviors that your organization is not ready for can lead to a ton of trouble. The proper and safest place to start is with Product/Service Health, or effectiveness, metrics. This will allow you to build a rapport with the organization and improve your customer relationships.
By starting with effectiveness measures, you will also allow your organization to ease into using metrics with less fear, less uncertainty, and less doubt. The organization will still have some worries over what you"re introducing, but by starting with the customers" viewpoint, you mitigate most of the risk. You can also use these metrics to produce high returns-understanding and pleasing customers are really the foundation of long-term business success.
It is convenient that the best place to start is also the easiest and safest, while the other possible choices require more effort, damage control, and has the highest risks for failure.
The following are important points to remember: Future Health metrics require a readiness most organizations lack.
Organizational Health metrics speak to the workers" situation, which most are not ready to hear; and if there is a lack of trust, most workers are not ready to share.
The lack of trust that makes measuring Organizational Health difficult makes collecting accurate data on how well the business is run, Process Health, unlikely.
Product/Service Health metrics require knowledge of the customer-who the customer is, what they see as important, and what they expect.
Conclusion.
Figure 6-7. The Answer Key, with tiers and quadrants.
While the Answer Key (Figure 6-7) offered multiple options for metrics development, the right place to start is with Product/Service Health (effectiveness), the customer"s viewpoint. Not only is this the safest course of action, it is also the best since it builds a foundation for the other quadrants in the third tier. If you create a working, and thereby useful, metrics program around the effectiveness of your product and service health, you will build the trust necessary to branch into efficiency metrics.
Once you"ve tackled the Return vs. Investment quadrants, you"ll have a basis of maturity preparing your organization for the State of the Union quadrants. The conclusion is as follows: Start with the customer view (effectiveness measures). Take your time and fully realize the benefits of this level of metrics. Let it become an accepted part of the organization"s culture.
Use the business view (efficiency measures) sparingly and when needed to investigate further into ways to improve effectiveness. (I"ll provide more insights into this later.) After you have built a healthy level of trust within the organization and a successful effectiveness-based metrics program, branch into other quadrants.
While management will want to go to the business view next, except for those measures necessary to improve effectiveness, I recommend skipping efficiencies and moving to the workers" viewpoint (Organizational Health). This will help build trust and result in greater and faster overall gains to the organization"s effectiveness. If the workforce is truly the greatest a.s.set-after developing metrics to improve the customers" experience, the next priority should be to improve the workforce"s. Address Future Health metrics primarily as a tool to determine the likelihood of achieving the organization"s goals. Then use them to determine if the goals were achieved. If you focus on the strategic plans portion of Future Health, you can create another safe haven for metrics.
In Chapter 11, Employing Advanced Metrics, I will go into more detail on the benefits and means for developing metrics in each of the third tier"s quadrants. For now, it is enough to promote Product/Service Health as the correct starting point and to stress the need to wait on the other areas until the time is right.
Triangulation.
Essential to Creating Effective Metrics.
And so it was, Momma mouse gave birth to three beautiful, but blind, mice. Once they were of age, the farmer"s wife took pity on them and decided to help them survive in the world. A world made much more dangerous due to their lack of sight.
The farmer"s wife decided it would be helpful if the mice learned which animals were friendly, and which ones might prefer to eat them than to partake in a visit. Since she was originally a city girl, she packed the three mice into her minivan and headed off to visit the museum of natural history. She thought using the exhibits would be safer than introducing them to the real things.
Although there weren"t too many African wild animals on the farm where they lived, the farmer"s wife thought it would be better to be "safe" than sorry.
They started with the tiger. She reasoned it was just a bigger version of the barnyard cat. The first mouse was on the back left paw. "It"s got claws and big feet," he shouted. The second mouse was on the tail. "It"s got a long strong tail." The third mouse, the oldest by three seconds, was at the face. "It has big eyes, big teeth, and whiskers."
The farmer"s wife smiled. "So, what is it?"
"A very big cat" they all said in unison.
"Very good. How about this one?" They all went over to the next animal.
"It"s not an animal at all. It"s a tree trunk," said the first mouse.
"No, it"s a snake hanging from a tree," said the second.
And the third, again at the face of the animal, said, "It"s a giant bird from the feel of the powerful wings."
"Put all of those together children," the farmer"s wife advised.
They thought for a bit and finally shouted together in perfect harmony, "An elephant!"
"Very good."
They continued this way through the animal exhibit, identifying wolves, dogs, lions, and bears. They learned about all types of animals, domestic and wild. Finally, they came to one they could not solve. It seemed to have the head of a lion, the body of a goat, and a snake"s tail.
"We"ve looked over the animal from head to toe and back again," they all agreed, "and we can"t agree on what it is!"
"That"s a chimera" said the farmer"s wife. "I am very impressed by you three. With an understanding of animal physiology, you have successfully identified every creature, using very different inputs."
The three mice beamed.
"But, we didn"t get them all right," said the third mouse, thinking of the chimera.
"No, but even with varying, complementary, solid data from multiple sources, you can"t always get to the "truth."" She was very pleased with how well they had done. Sometimes it"s just not possible to derive the correct answer, even with a lot of data. Of course, it helps if you can observe the item yourself.
Researchers are the ultimate data users.
Triangulation-a Historical Perspective.
The mice were forced to use triangulation to identify the animals. They compared different pieces of information gathered using different methods to determine the answers to their question (what is the animal that we are touching?). If the mice looked at only one piece of information, they"d have the wrong answers.
I bow to Norman K. Denzin, a professor of communications and sociology at the University of Illinois at Urbana-Champagne, for his using "triangulation" to mean using more than two methods to collect data in the context of gathering and using research data. My definition is much like it-using multiple measures, as well as collection methods, for processing the information used in a metric.
A major reason for using triangulation, according to Denzin, is to reduce (if not eliminate) bias in the research. By using triangulation we ensure that we have a comprehensive answer to the question.
In his book, Sociological Methods: A Source Book, Denzin describes the following four types of triangulation:1 Data source: Using multiple data sources.
Investigator: Using more than one person to collect, a.n.a.lyze, or interpret a set of data.
Theory: Using multiple perspectives to interpret a set of data.
Method: Using multiple methods to capture the data.
But even Denzin would have to agree that it was not his "invention" since he cites a paper by Eugene J. Webb in which Webb offers a triangulated approach to sociological research. Denzin offers that as you increase the number of differing measures used to provide insight on a single issue, the definitions move from an abstract thought, to validated concept, and finally to proven reality. Although Denzin may not have been the first to coin the term "triangulation," I like his explanation. It matches very closely to how I use the term in the context of a metrics program.
Denzin"s four types of triangulation aren"t an exact match to the different types that I use for metrics-and that is partly due to the nature of the use of our results. While Denzin is giving guidance for sociological research that has the purpose of finding deeper truths within his field and having those truths debated and challenged within the scientific community in journals and experiments-our needs (yours and mine) are much simpler and more practical.
______________________________.
1 Norman K. Denzin, Sociological Methods: A Sourcebook (Chicago: Aldine Publishing Co., 1970).
The need for different investigators is relatively unnecessary in my field, although it is critical in the field of research to avoid bias in the a.n.a.lysis. I agree with having different data sources, different perspectives, and different methods. I refer to perspectives as viewpoints. Let"s now look at my version of triangulation.
Triangulation as a Practical Application.
In 2003 I developed a concept of using multiple measures to construct a comprehensive metric. I was developing a metrics program for my IT Department and I needed to create a complete story that answered two questions posed by the top executive officer. His first question was: "How are you doing?"
He was inquiring about the health of our IT organization. A simple but driving question. It created not only a set of metrics, but it sent me on a journey that culminated in doc.u.menting many of the methods and theories that I"ve captured in this book.
The executive"s second question was a great companion to the first: "How do you know?"
I think I liked these questions more for their simplicity and directness than anything else. They drove our organization toward a viable, practical metrics program.
Triangulation of Measures.
I focused on the Effectiveness quadrant of the Answer Key (Figure 7-1), choosing to qualify the question as being from our customers" point of view-because the executive asking it was our top-level customer. While the executive may have cared about how efficiently we were conducting our affairs, his first primary concern was how well we were serving our customers.
Figure 7-1. The Answer Key.