Here is the five-level evaluation framework that Jack developed. The graphic also shows the measurement focus and typical measures for each level.

Level 3 is where participants return to their workplace and apply what they learned at Level 2 and agreed to do at Level 1. The measurement focus and typical measures at this level want to determine if participants have made what they learned a part of their daily practice with their jobs. The participants may have learned the content, but if they don’t use it, nothing will change. This is why we also look for anything that might prevent the participants from applying what they learned, or if they have what is necessary to change their behaviors. We call these barriers and enablers.

If we completed our first three steps correctly, we should have designed a program that will transfer to the workplace and make a positive difference. It is the most difficult to collect Level 3 and 4 data, so we must make sure we have done a good job of planning how we will collect it and who will help us collect this data.

Make it stick focuses on two types of data that are collected after a program is implemented: application of what was learned after participants return to the workplace and the impact these changed behaviors have on the business need measures.

Make it Stick: Design for Application and Impact

This is where most programs fail in delivering results. Participants go through the course and they demonstrate that they learned the knowledge or skills and are committed to using what they have learned, but something prevents them from actually using it in the workplace. What we must do is figure out how we can design our program to overcome these obstacles, and then evaluate after the program to see how well we did with helping the participants overcome those obstacles. We must explore different options and experiment, instead of doing things the same way we have always done them. How do we do that? We start by focusing on the objectives. We focus on the objectives that the stakeholders want us to improve. This is how success will be defined. We must figure out how to achieve those objectives. We must also figure out how to later evaluate to determine how well we did?

We must ensure that participants are applying what they learned on the job. We must also find the barriers that we must work around ahead of time and solve or eliminate them. We also must get the buy-in of various stakeholders to help us do that. The stakeholders will have a vested interest in helping us solve these problems and achieve the objectives that they helped us write. Getting them to write those objectives and working with them to address the needs affecting the objectives will help us earn their buy-in.

Level 3, Application is where supervisors live. Supervisors are responsible for getting the work that they report to their bosses done. Level 4, Impact is the data that executives want to see. If we do our job during the needs assessment, by recording their objectives and how they define success, we should have their help to collect our evaluation data because those measures are what they want to see. It helps for them to know how well the program achieved their objectives. We plan for that upfront using our data collection planning document, which we also used earlier for Levels 1 and 2. This is when we decide what data collection methods or instruments we will use and whether stakeholders will support them.

Data Collection Methods

This table shows some common methods of data collection. You will notice that some methods for data collection at Levels 1 and 2 are also used at Levels 3 and 4. These are great options if we want to save money. We could use a questionnaire to collect data at all four levels. We can collect raw data from Level 2 to see how well the participants are applying what they learned. With observation, at Level 3, participants cannot know they are being observed. If they know they are being observed, then it is considered a Level 2 evaluation. Interviews and focus groups are a great way of collecting data to find out how the application is going and what barriers or enablers are you encountering. In these sessions, you can ask, “Is there something preventing you from applying what you learned? Is there anything that will help you apply what you’ve learned?” Our focus with data collection is to determine how well we are moving towards achieving those Level 3 objectives, which are going to improve workplace performance resulting in improved impact measures.

Action Plan

Action plans or performance contracts are collected at Levels 3 and 4. Below is an example of an action plan. In this case, the objective was to eliminate gel waste and improve quality. The participant also wanted to decrease the amount of waste by 80%. The question was what is this individual going to do to meet this objective? This individual noted five actions to change behavior and achieve those objectives. The participant wrote, on the right-hand side, what was expected to happen as a result of changing those behaviors. At the bottom, some of the intangibles, barriers, and enablers are listed. At Level 3, we are collecting data to find out how well we did at achieving those objectives? We can also use an action plan to look at Level 4 objectives.

Selecting Methods

When we select our data collection methods, we must consider the following:

Make sure your participants will have the time needed to do what you ask. We have all been in situations where we are doing our jobs, and somebody asks us to fill out a survey or someone is collecting data about something, and we just do not have time to participate. Let’s make sure that we give participants enough time to help us collect the data during the planning phase. Have the same respect for the supervisors’ time. Be sensitive to their time and communicate with them ahead of time to get their buy-in.

Consider the cost of the data collection method. What is it going to cost to collect the data? Do not disrupt the workflow with what you are doing. You always want to minimize the disruption, otherwise, participants will not cooperate. Be as accurate with the questions as possible so that the answers collect the data you need about each objective. This is not a science project, so be as accurate as necessary to ensure stakeholders find the data credible. Utility addresses the idea of use. If you are not going to use the data, do not collect it. Finally, consider the culture and philosophy of the organization. During the planning phase, you want to find out what the organization will or will not support for data collection. They may have a policy of no surveys or questionnaires, or the organization may not allow focus groups. Find out ahead of time what the stakeholders and organization will agree to.

Data Sources

The last three questions to answer and plan for are important. You should bring in various stakeholders to help you to answer these questions. To ensure that you are finding credible people, identify the credible data sources in your organization when you prepare your data collection plan. That way you can easily identify the credible sources at the time of data collection. In the example below, you can see we did just that.

If you see these people in the hallway where you work, you know they are credible. Don’t forget the participants. Participants are very credible sources of data collection at Levels 3 and 4. Supervisors are also a good data source as well, but sometimes they are not as close to the data. Use them as a backup. Direct reports and the people that report through these individuals can provide you with good data to see if participants have improved their behaviors in the workplace. The people that work with the participants, their peer group, will know if they have changed behaviors and if there has been improvement. You may also have internal staff that can provide data as well as external sources like customers. Finally, look at organizational records. Organizational records are a great source for determining improvement.

When is the Best Time to Collect Data

When is the ideal time to collect the Level 3, Application data? Supervisors are a great resource to answer this question. They will know when to expect the behavior change. Ask them during the planning phase. Do the same for Level 4, Impact measures. Supervisors will know when to report this information. Ask them when they normally report this data and use that timeframe for your data collection. You also want to make sure that when you are collecting the data, it is convenient for everybody and without time constraints. That conversation depends on the program, and the key is, you are not making this call yourself. You are working with the people who will know when to collect this data.

When you are choosing your methods and determining how you are going to collect the data, balance the benefits of the data with the costs.