Here is the five-level evaluation framework that Jack developed. The graphic also shows the measurement focus and typical measures for each level.

Level 1 measures the reaction and planned actions of the program participants. We want participants to perceive the program as relevant, important, useful, and other measures you see listed in the right column. If these measures are positive, we can predict there is a higher likelihood the participants will change their workplace behaviors, which should lead to improved business measures.

If participants have a negative reaction, they won’t learn what is necessary to change their workplace behaviors, and nothing will change in the workplace. This is not good because workplace change is essential to improve the business measures that executives expect to see improved in return for the resources they committed to the program.

We want the program to matter to all participants and stakeholders. To do this, we must put ourselves in their shoes. We want to develop programs with content that is relevant, meaningful, and important to the individuals and to the organization, making the program something that they will use. We also want to provide participants with examples, activities, and exercises that reflect what they are learning, what they will do with what they have learned, and the impact it will have on the organization.

How do we make it matter? Five techniques are listed:

Focusing on developing objectives during the Business Alignment V Model stage defines how the stakeholders see success. We also must think about the return on the investment. This can be the financial return, or it could be intangible measures. We must ensure what we do is relevant to the participants. The participant’s perception of the program is important, and they will commit to act because of it. There is nothing worse than going to a program of some type and finding out it has nothing to do with your job or it cannot help you.

We cannot design programs that matter if we do not define the needs of the participants and how they will define success. We do this with objectives. We use these objectives to evaluate the program and learn if it mattered to the participants. We plan how we will evaluate these levels using a planning document.

The data collection plan is categorized into five levels. We have a place where we can record the broad program objectives at each level. The third column lists the measures. This form is our Business Alignment V Model. With this, we want to identify what methods or instruments we are going to use to collect the data. We also must figure out what the data sources are, the timing of when we will collect the data, and who is responsible for collecting it. We complete this form during the planning phase to ensure we get the stakeholder buy-in and their support from the very beginning. The last thing we want to do is plan to use a data collection technique and then management comes back to us saying they are not supporting the program because we did not plan it with them.

We are going to be focusing on the first two levels. Surveys are typically what we find at the end-of-course evaluation. We call them “smile sheets.” We can use surveys to ask them questions about what participants have learned. Questionnaires are great for collecting raw data. Questionnaires are used when we want to go beyond participant attitudes and their perceptions about the course. We collect raw data by asking them questions about what they learned and how they plan to apply it.

Observations occur when we watch participants applying what they learned. This technique can be completed by the instructor simply watching to see if participants are applying what they learned. Interviews and focus groups typically occur as a follow up after the course is completed. This technique is used to determine a deeper understanding of how participants are using what they learned and obtaining their perceptions of what they learned. Finally, the last three techniques deal with test questions, demonstrations, and simulations. These are done for Level 2, Learning only, either through demonstrating knowledge on tests and quizzes or actually demonstrating the new skills or knowledge.

Work with management to select the most appropriate technique and define the most appropriate method to collect the data necessary to answer the questions that have been defined to determine if the objectives have been met.

After the data collection method is defined, we must determine who is going to collect the data from the data sources, when is the most appropriate time to collect that data, and who is responsible for collecting that data. We want to collect data during implementation because it is much easier to collect the data when participants are under our control. It may be more difficult to collect data after the participants leave the learning program.