Effectively Using Mystery Shopping to Enhance the Customer Experience | by Randall Brandt | Maritz Research
Mystery shopping is one of many techniques used to develop an understanding of the customer experience. Persons who are either real customers, or who fit the desired customer profile, go through actual product or service encounters. Afterwards, these "shoppers" systematically record the incidence of specific events or customer contact associate behaviors, and often make additional measurements or observations regarding the product/service experience as well. In essence, be they genuine or potential customers, mystery shoppers are viewing the product/service experience from the "customer's side of the table."
Mystery shopping can provide a variety of benefits and insights, including:
- Enabling an organization to monitor compliance with product/service delivery standards and specifications
- Enabling marketers to examine the gap between promises made through advertising/sales promotion and actual service delivery
- Mystery shopping is useful in monitoring the impact of training and performance improvement initiatives on compliance with or conformance to product/service delivery specifications
- Identifying differences in the customer experience across different day parts, locations, product/service types, and other potential sources of variation in product/service quality
Mystery shopping also has some important limitations:
- Limited projectability – traditionally, while mystery shoppers are viewing product/service experiences from the customer's side of the table, shoppers may not be genuine customers.
- The criteria used by mystery shoppers to monitor the quality of product/service encounters will only be useful in effective management of customer experiences to the degree that these criteria are based upon actual customer expectations, needs, and requirements.
- Continuous efforts to keep criteria consistent with evolving customer needs and requirements must be undertaken.
Understanding the strengths and weaknesses of alternative methods of capturing the voice of the customer – and how they impact an organization's ability to address alternative managerial objectives and applications – is a key to building a comprehensive and effective customer experience management system. If mystery shopping is to be a part of that system, there are several methodological considerations that must be addressed. In particular, the following issues should be considered carefully in designing and executing a mystery shopping program:
1. Selection of the attributes and evaluative criteria used by the mystery shopper
Ensure that the attributes/criteria used by mystery shoppers reflect things that are important to customers. What do customers want or expect from the product/service experience? What does the customer see, experience, scrutinize, and evaluate? Results are more likely to enable the organization to take actions that actually improve customer satisfaction and loyalty.
The relative importance of the attributes also should be taken into account. This will ensure that any weighting scheme used to compute mystery shopping scores accurately reflects the differential impact of each attribute, and gives the greatest weight to the attributes that are most influential in determining overall customer satisfaction and loyalty.
Measures and criteria used by mystery shoppers should be aligned with those used in surveys, comment cards, and other methods of gauging the customer experience, so that convergence of findings obtained from these different data sources may be determined. In some instances, this means making an effort to match mystery shopping measures/criteria with those used in other methods. This makes it possible to determine if results of mystery shopping point to the same "action items" indicated by results obtained from surveys, inbound customer communications, and other data sources. Such convergence, when established, tends to "validate" conclusions regarding priorities for effectively managing and improving customer experiences.
In some cases mystery shopping measures and criteria cannot or should not be matched those used in other methods. However, an effort should at least be made to ensure that measures may be meaningfully linked based upon relevance or interdependence. For example, in industries such as banking, retail shopping, and hospitality, "waiting time" is an element that commonly appears in both customer surveys and mystery shopping instruments. The key is to ensure that results obtained from each of these data sources may be linked. If a mystery shopper gauges actual wait time, while surveys ask customers to report how long they perceived to have waited in line, data analysis should focus on how customer perceptions vary with increases or decreases in actual wait time.
Alternatively, it can be equally insightful to also ask both mystery shoppers and real customers, if their wait time was "satisfactory." Not only does such information make it possible to "calibrate" shopper and customer perspectives, but it also enables the organization to set operational performance targets for wait time that are most likely to translate into positive customer experiences.
2. Adequate sampling or coverage of events and/or sites via mystery shopping
As is the case with customer and market surveys, the quantity and frequency of observations directly impacts an organization's ability to draw projectable conclusions from mystery shopping results. Results are more likely to be "representative" if a bank branch or customer service center is "shopped" on multiple occasions within a relatively tight timeframe, than if that branch/center is shopped once within a relatively wide and arbitrary timeframe. This is another way of saying that the results of mystery shopping are subject to the effects of both sampling and non-sampling error. Therefore, steps must be taken to ensure adequate sampling or coverage of the events, channels, and/or sites about which conclusions will be drawn based on mystery shopping.
Some specific steps that may be taken to minimize sampling error and/or bias in mystery shopping results include:
- Carefully defining the entity or entities for which statistically accurate conclusions must be formulated, and developing sample size requirements at the entity level. This becomes especially critical when results are going to be used in a manner that demands a high degree of statistical precision (e.g., if store managers' compensation is to be based at least partially on results of mystery shopping).
- Using results of mystery shopping as one of several performance indicators regarding the customer experience. When results of mystery shopping, customer surveys and customer comment cards all point to the same areas for improvement, managers are less likely to question the accuracy or validity of any single data source/method.
- Matching the timing, location, and/or frequency of mystery shopping to actual customer behaviors, channel usage, and related patterns. To illustrate, in the case of so-called "quick service restaurants," if 60% of business is conducted at the drive-through window, while 40% comes from walk-in customers, an attempt should be made to allocate the total number of mystery shops to be conducted in a manner that mirrors these proportions.
Execution of the preceding steps probably won't eliminate sampling error and/or bias altogether, but it will go a long way toward minimizing the impact of such error/bias.
3. Reliability and consistency of the mystery shopping criteria and procedures
The mystery shopping method must be reliable. This means: (a) providing shoppers with clear and explicit guidelines for using mystery shopping criteria; (b) ensuring that mystery shopping criteria and procedures are applied consistently by each shopper from one site/event to the next; and (c) ensuring that the criteria and procedures are used similarly by different shoppers.
There are well-established methods for determining the reliability of mystery shopping procedures, and these should be applied in a pilot test of the shopping method before its full-scale implementation. For example, suppose that shoppers are asked to observe whether a store has floors that are "clean and free of debris." This item should be supported by clear definitions and descriptions (including photos or illustrations, if available), along with guidelines that explain any allowances and thresholds. Similar support should be provided for all measures and criteria to ensure that shoppers are as objective and consistent as possible across sites and events.
It is also critical to establish consistency among shoppers. This is achieved by having multiple shoppers evaluate a common set of events or transactions to establish correlations and levels of agreement. For example, in the case of a call center, multiple shoppers should listen to and evaluate a common set of recorded calls. Results may then be compared and, if necessary, efforts undertaken to maximize consistency and agreement among shoppers. This process, sometimes referred to as "calibration," is another way to ensure the reliability and consistency of mystery shopping procedures.
All of the steps above promote consistency and objectivity in mystery shopping procedures. A cautionary note, however, is in order: Emphasizing evaluative criteria and/or aspects of product/service delivery that are readily defined - or that reflect the organization's priorities - facilitates objectivity and consistency. However, too much emphasis on these elements may preclude shoppers from recording unique or incidental things that might furnish important insights and opportunities for product/service improvement. Therefore, it is critical to "leave room" in the shoppers' experiential checklist for these additional elements and insights. The result will be a mystery shopping approach that is disciplined and reliable, but also opportunistic.
4. Keeping shoppers and customers connected
Mystery shopping enables an organization to evaluate the degree to which product/service delivery complies with performance targets, standards, specifications, policies, and desired behaviors, that are intended to ensure a positive customer experience. Presumably, such targets, standards, specifications, and policies are established on the basis of a clear understanding of customer needs, expectations, and requirements. However, customer needs, expectations, and requirements are moving targets that continually change and evolve over time. Therefore, it is critical to check periodically the alignment of performance standards, etc. with changing customer expectations, and to make adjustments in mystery shopping criteria and procedures accordingly. This will increase the odds that shoppers and customers "stay connected."
Summary and Conclusion
There is no "perfect" method of learning all that is needed to manage customer experiences effectively. Thus, it is critical to employ and integrate multiple methods of measurement and observation to develop a comprehensive and panoramic perspective.
While it does not always capture the voice of the customer directly, mystery shopping can be designed and implemented in ways that make it a very good method of customer-driven quality assurance. Generally speaking, mystery shopping is most useful for gaining insights that enable an organization to address two key questions about the customer experience:
- Do we actually deliver products and services in ways that are consistent with our performance goals, standards, specifications, and/or policies?
- Do we actually deliver products and services in ways that are consistent with promises and claims made in advertisements, promotions, and other customer and market communications?
By addressing the methodological considerations discussed above, an organization can go a long way toward ensuring that mystery shopping fulfills this purpose.
D. Randall Brandt | As Vice President, Customer Experience and Loyalty Research, Brandt leads Maritz Research’s customer experience and loyalty research efforts and works with Maritz’s other business units to help clients integrate research with performance improvement efforts. Prior to joining Maritz Inc., Brandt worked for Burke, Inc. as senior vice president and practice leader for both customer loyalty & relationship management and linkage & integrations services. His research, consulting and marketing expertise has assisted clients in more than 200 firms on four continents during the past 19 years. He received a bachelor’s degree in communication from Purdue University and a master’s degree in communication from Indiana University. He also earned his doctorate in communication and psychology at Michigan State University. Brandt served on the Board of Examiners for the Malcolm Baldrige National Quality Award for five years, and on the Human Resources Task Force of the Advisory Council for the United States Environmental Protection Agency for two years.
Published by Maritz Research
Date: Volume 18 - July 2005
Subscribe to the Maritz Research Forum
About Maritz Research
As one of the world's largest marketing research firms, Maritz Research, a unit of Maritz, helps many of today's most successful companies improve performance through an actionable understanding of their customers, employees, and channel partners. Founded in 1973, Maritz Research offers a range of strategic and tactical solutions concentrating primarily in the automotive, financial services, hospitality, telecommunications and technology and retail industries. The company has achieved ISO:2 0252 registration, the international symbol of quality. Maritz Research is a member of CASRO and official sponsor of the American Marketing Association.
