Here is the five-level evaluation framework that Jack developed. The graphic also shows the measurement focus and typical measures for each level.

At Level 4, we want to measure and learn what impact the program had on the business measures that executives want to see improved. It is important to note that we do not claim our program was responsible for 100% of the improvement unless we have credible evidence that supports this claim. This is why we isolate the effect of our program, and take credit for impact improvements that we can only rightfully claim, and give credit for improvements resulting from other programs.

The monetary values will be found in the data analysis process phase. We start by isolating the effect of our program so we can protect our credibility when we claim the impact of our program. This is such an important step that it has its own guiding principle. There are people who argue that you cannot isolate the effect of a program. We will show you eight techniques used by over 6,000 organizations around the globe which have proven you can.

Make it stick focuses on two types of data that are collected after a program is implemented: application of what was learned after participants return to the workplace and the impact these changed behaviors have on the business need measures.

Make it Stick: Design for Application and Impact

This is where most programs fail in delivering results. Participants go through the course and they demonstrate that they learned the knowledge or skills and are committed to using what they have learned, but something prevents them from actually using it in the workplace. What we must do is figure out how we can design our program to overcome these obstacles, and then evaluate after the program to see how well we did with helping the participants overcome those obstacles. We must explore different options and experiment, instead of doing things the same way we have always done them. How do we do that? We start by focusing on the objectives. We focus on the objectives that the stakeholders want us to improve. This is how success will be defined. We must figure out how to achieve those objectives. We must also figure out how to later evaluate to determine how well we did?

We must ensure that participants are applying what they learned on the job. We must also find the barriers that we must work around ahead of time and solve or eliminate them. We also must get the buy-in of various stakeholders to help us do that. The stakeholders will have a vested interest in helping us solve these problems and achieve the objectives that they helped us write. Getting them to write those objectives and working with them to address the needs affecting the objectives will help us earn their buy-in.

Level 3, Application is where supervisors live. Supervisors are responsible for getting the work that they report to their bosses done. Level 4, Impact is the data that executives want to see. If we do our job during the needs assessment, by recording their objectives and how they define success, we should have their help to collect our evaluation data because those measures are what they want to see. It helps for them to know how well the program achieved their objectives. We plan for that upfront using our data collection planning document, which we also used earlier for Levels 1 and 2. This is when we decide what data collection methods or instruments we will use and whether stakeholders will support them.

Data Collection Methods

This table shows some common methods of data collection. You will notice that some methods for data collection at Levels 1 and 2 are also used at Levels 3 and 4. These are great options if we want to save money. We could use a questionnaire to collect data at all four levels. We can collect raw data from Level 2 to see how well the participants are applying what they learned. With observation, at Level 3, participants cannot know they are being observed. If they know they are being observed, then it is considered a Level 2 evaluation. Interviews and focus groups are a great way of collecting data to find out how the application is going and what barriers or enablers are you encountering. In these sessions, you can ask, “Is there something preventing you from applying what you learned? Is there anything that will help you apply what you’ve learned?” Our focus with data collection is to determine how well we are moving towards achieving those Level 3 objectives, which are going to improve workplace performance resulting in improved impact measures.

Action Plan

Action plans or performance contracts are collected at Levels 3 and 4. Below is an example of an action plan. In this case, the objective was to eliminate gel waste and improve quality. The participant also wanted to decrease the amount of waste by 80%. The question was what is this individual going to do to meet this objective? This individual noted five actions to change behavior and achieve those objectives. The participant wrote, on the right-hand side, what was expected to happen as a result of changing those behaviors. At the bottom, some of the intangibles, barriers, and enablers are listed. At Level 3, we are collecting data to find out how well we did at achieving those objectives? We can also use an action plan to look at Level 4 objectives.

Selecting Methods

When we select our data collection methods, we must consider the following:

Make sure your participants will have the time needed to do what you ask. We have all been in situations where we are doing our jobs, and somebody asks us to fill out a survey or someone is collecting data about something, and we just do not have time to participate. Let’s make sure that we give participants enough time to help us collect the data during the planning phase. Have the same respect for the supervisors’ time. Be sensitive to their time and communicate with them ahead of time to get their buy-in.

Consider the cost of the data collection method. What is it going to cost to collect the data? Do not disrupt the workflow with what you are doing. You always want to minimize the disruption, otherwise, participants will not cooperate. Be as accurate with the questions as possible so that the answers collect the data you need about each objective. This is not a science project, so be as accurate as necessary to ensure stakeholders find the data credible. Utility addresses the idea of use. If you are not going to use the data, do not collect it. Finally, consider the culture and philosophy of the organization. During the planning phase, you want to find out what the organization will or will not support for data collection. They may have a policy of no surveys or questionnaires, or the organization may not allow focus groups. Find out ahead of time what the stakeholders and organization will agree to.

Data Sources

The last three questions to answer and plan for are important. You should bring in various stakeholders to help you to answer these questions. To ensure that you are finding credible people, identify the credible data sources in your organization when you prepare your data collection plan. That way you can easily identify the credible sources at the time of data collection. In the example below, you can see we did just that.

If you see these people in the hallway where you work, you know they are credible. Don’t forget the participants. Participants are very credible sources of data collection at Levels 3 and 4. Supervisors are also a good data source as well, but sometimes they are not as close to the data. Use them as a backup. Direct reports and the people that report through these individuals can provide you with good data to see if participants have improved their behaviors in the workplace. The people that work with the participants, their peer group, will know if they have changed behaviors and if there has been improvement. You may also have internal staff that can provide data as well as external sources like customers. Finally, look at organizational records. Organizational records are a great source for determining improvement.

When is the Best Time to Collect Data

When is the ideal time to collect the Level 3, Application data? Supervisors are a great resource to answer this question. They will know when to expect the behavior change. Ask them during the planning phase. Do the same for Level 4, Impact measures. Supervisors will know when to report this information. Ask them when they normally report this data and use that timeframe for your data collection. You also want to make sure that when you are collecting the data, it is convenient for everybody and without time constraints. That conversation depends on the program, and the key is, you are not making this call yourself. You are working with the people who will know when to collect this data.

When you are choosing your methods and determining how you are going to collect the data, balance the benefits of the data with the costs.