Evaluating to level one is not simply a question of handing out a set of smile sheets, Godfrey Parkin offers some alternatives.
Whenever I am involved in an evaluation project, I advocate getting rid of the smile sheet completely, and replacing that tortured questionnaire with one closed question, plus an open follow-up to encourage respondents to reveal what really matters to them: “Would you recommend this course to a friend or colleague? Why or why not?”
The response tells you unambiguously about the level of satisfaction of the learner, and any clarification offered tells you about the issues that really matter to that learner. That’s more than is called for at Level 1.
It’s not always possible to reduce things to one question, but I see it as a starting point in the negotiation. I tend to be somewhat dismissive of Level 1 evaluations. That is not because they serve no purpose (they are vital), but because they attract way too much attention at the expense of business impact studies, and because they are often poorly designed and inaccurately interpreted.
Every training intervention needs some kind of feedback loop, to make sure that – within the context of the learning objectives – it is relevant, appropriately designed, and competently executed.
At Level 1 the intention is not to measure if, or to what extent, learning took place (that’s Level 2); nor is it intended to examine the learner’s ability to transfer the skills or knowledge from the classroom to the workplace (Level 3); nor does it attempt to judge the ultimate impact of the learning on the business (Level 4). Level 1 of Kirkpatrick’s now somewhat dated “four levels” is intended simply to gauge learner satisfaction.
Typically, we measure Level 1 with a smile sheet, a dozen Lickert-scaled questions about various aspects of the experience. At the end of the list we’ll put a catch-all question, inviting any other comments. I won’t repeat the reasons why the end-of-course environment in which such questions are answered is not conducive to clear, reasoned responses. But the very design of such questionnaires is ‘leading’ and produces data of questionable validity, even in a calm and unhurried environment.
Far too many of the smile sheets that I see put words or ideas into the mouths of learners. We prompt for feedback on the instructor's style, on the facilities and food, on the clarity of slides. The net effect is to suggest to respondents (and to those interpreting the responses) that these things are all equally important, and that nothing outside of the things asked about has much relevance.
By not prompting respondents you are likely to get to those things that, for them, are the real burning issues. Open questions are not as simple to tabulate, but they give you an awful lot to chew on.
Now the one-question approach does not necessarily give you all the data that you need to continuously fine-tune your training experience – but neither does the typical smile sheet. Trainers need to understand that sound analytical evaluations often require multi-stage studies. Your end-of-course feedback may indicate a problem area, but will not tell you specifically what the problem is. A follow-on survey, by questionnaire, by informal conversation, or by my preferred means of a brief focus group, will tell you a great deal more than you could possibly find out under end-of-course conditions.
The typical smile sheet is a lazy and ineffective approach to evaluating learner satisfaction. It may give you a warm and comfortable feeling about your course or your performance as a trainer, or it may raise a few alarm flags. But the data that it produces is not always actionable, is rarely valid, and often misses the important issues.
In market research, or any statistical field for that matter, there are two important errors that good research tries to mitigate. Known as Type One and Type Two Errors, they measure the likelihood of seeing something that is not there and the likelihood of missing something important that is there. I have never heard anyone address these error types in their interpretation of Level 1 results. We see in our smile-sheet results what we want to see, and react to those things that we regard as relevant. If we are so smug in our knowledge that we know what is going on anyway, why do we bother with token smile sheets at all?
Godfrey Parkin , 10 August 2005 @ 19:14 PM It's only a label
Leslie -- thank's for your comments. Kirkpatrick’s “Levels” are simply a somewhat outdated framework for labeling the nature of what is being measured. Level One does exist. It exists in the evaluation literature, and in the heads of trainers everywhere, and in the filing cabinets stuffed with reactionnaires in every training office on the planet. It is merely a gauge of trainee reaction, and should not pretend to be anything more than that. But it is both necessary and valid to attempt to measure that reaction, as part of your quality assurance process.
I think if you ‘revisit’ my comments in past columns, you’ll see that I agree with you that participant satisfaction surveys are not a valid measure of training effectiveness, let alone of business impact. Yet they are often interpreted as such by the majority of organizations – hence my own intermittent rants on the subject.
Leslie Rae , 10 August 2005 @ 09:08 AM Reactionnaires
Godfrey Much as I enjoy reading your comments, I must take you to task over helping the perpetuation that there is such a thing as 'Kirkpatrick's Level 1' in evaluation. The completion of a reactionnaire, however well constructed has got absolutely nothing to do with evaluation, which is only about the measure of learning achieved against the agreed objectives and its subsequent successful evaluation. Reactionnairesgive only what their name suggests - the gut feeling about their feelings, opinions, moods, attitudes, and all other things completely subjective. This is not evaluation and I wish intelligent consultants and trainers would stop, as I say, helping to perpetuate the nonsense. OK they have a place, when needed, and if good re'trres can be used to get more money out of the organization and encourage learners to come - but that is not evaluation. Evaluation (or rather validation when refferring to end of course instruments) is about learning and all efforts by trainer and learner should be to determining that end. I would suggest reference to my best-selling book on the subject 'Assessing the Value of your Training: The Evaluation Process from Training Needs to the Report to the Board', Gower 2002 Leslie Rae PS So please do not 'Revisit Kirkpatrick's Level One' it doesn't really exist! Leslie Rae
paul cook , 09 August 2005 @ 22:25 PM Combine level 1 & 2
By dividing delegates into small groups and then asking each group to reflect on a particular day of learning directly on close and present results the following morning has proved useful. This can include any level 1 and, 2 type responses by getting delegates to identify their own learning.The `Least helpful`, Most useful`, `How I would improve the day` and `What I have learnt` questions would be the basic framework to build on.
Alternatively, or in order to drill down on issues highlighted by group feedback, a `Structured Debrief` at the end of the course or a particular theme, using the structure and ethics of this process actively involves all delegates and gives a huge amount of Level 1 and 2 responses.
Michael Brown , 09 August 2005 @ 10:23 AM Agree completely, and even so people need help with how to use Level 1
Thanks Godfrey for another welcome jab in the ribs! As a provider it has sparked off an internal debate about what to measure on the day. In my experience, people struggle with what to do with information. Apart from the paper mountain it generates (there is a solution, by the way, it's called a digital pen), what is the most meaningful use of it? As training providers we use it for looking at trends over 6 to 12 months in a particular course, and for comparing performance between delivery media and between trainers. Oh, and of course for putting out very rare fires. What other uses do people make of the data? Michael Brown
Keith Cheetham , 09 August 2005 @ 03:06 AM Happy Sheets Yes or No
I find this argument very interesting from a trainers point of view in that most of us work for a boss and that boss wants to know how effective "Your" training sessions are, hence the happy sheets, I personally conduct a Rocks & Roses Session before the happy sheets and let the participants voice what they liked or disliked about the presentation, what they would keep and what they would discard type of questions and they answer what they think not ticking the box for whatthey think "You" want to hear.
Godfrey Parkin , 08 August 2005 @ 15:03 PM If it's not good, why are you running it?
Doug,
I agree that skillful questioning is essential. But it is my belief that much of what is considered valuable structured formative feedback should have been collected and processed (and acted upon) at the alpha, beta, and pilot stages of a course roll-out. Even if you don't have the time to test before roll-out, by the time a course is in its fourth or fifth live session, you should know it is good, or you should not not be running it. And if you know it's good, why keep asking those formative questions? Godfrey Parkin
Michael Mallows , 08 August 2005 @ 14:58 PM KLP Personal Insight Apply
Apart from the usefulness or otherwise to the client and the trainer (me!) of level 1 feedback, another reason that I use it is to give participants a reinforcing recap of the training.
Whether or not there are happy sheets, I might ask people to think of and give brief feedback on (at least) 1 Key Learning, a personal insight, and (at least) one thing they intend to put into practice that will manifest in other people's senses. I often alert people to this format at the beginning of the training / day, and use it, or variations of it, as the day progresses e.g. after an exercise. It needn't take very long. Even 20 delegates, working in pairs, could do such an exercise in a few minutes. Less if, say, they are asked to 'comment on something in no more than 5 words'. The feedback is for not my benefit, so it is not necessarily the case that I need to hear or know it.
Doug Smith , 08 August 2005 @ 13:49 PM They need not be smile sheets
I find tremendous value in skillfully worded Level 1 evaluations. The form at the end of each class need not be a "smile sheet" at all. While I have seen some trainers manipulate their activities to elevate the scores of their evaluations, if the form is distributed early enough and clearly tied to the learning objectives and purpose of the program it is possible to collect valuable and immediately useful feedback.
Sometimes I think that we as trainers are too hard on ourselves and the tools that we use. It isn't the tool that's flawed, it is the application.
Select your questions with the intention of collecting useful feedback and identifying how effective your program has been and I think that you will see the value in the evaluation.
While I do like your question, "would you recommend..." it falls short of the robust feedback that is available just for asking.
John Hughes , 08 August 2005 @ 12:22 PM Smile sheets
I agree entirely with the view that smile sheets/happy sheets, call them what you will, are often less than helpful. From memory, most are focused on "basic needs": food and comfort!
However, I wonder if identifying NQF levels would have been more helpful overall than "outdated" ones.
Nik Kellingley , 08 August 2005 @ 10:04 AM Here, Here
I'm in agreement here. I think the smile sheet has become an awful add on to most training days - if the delegates are unhappy with facilities, lunch etc. then they can write that in the "anything else?" section - not much of a metric but who cares. I used four questions in regard to the last course I had delivered - relating to training materials, course, trainer and product (it was a software training event) and that's it. I'm not sure I'd go as far as narrowing that down to one but certainly a minimal amount is better than the other approach - my CIPD course had a questionnaire with almost 20 fields to rank out of 10 and 6 free text fields as well!
Graham O'Connell , 05 August 2005 @ 12:58 PM Level 1 evaluation
Godfrey has raised some valuable points about level 1 evaluation. I particularly agree with the idea of a goal free question (based on the work of Scriven). I am not sure that is all I would use but it is certainly an under-rated tactic in my view. I would add that level 1 evaluation - getting immediate feedback from customers - has three very valuable purposes. Firstly, it is a useful way to assess the quality of the learning process and the learning experience. Secondly, it is a gauge of customer satisfaction. Word of mouth is such a powerful marketing tool and if you don't keep on top of customer satisfaction then you can easily undermine what might otherwise be an extremely useful learning event. Thirdly, having an end of event evaluation form shows that you are willing to learn too. If you as a trainer don't invite feedback, learn and move forward, what right have you got to expect others to? Much of the literature on evaluation these days seems to miss the small but important role of level 1 evaluation because it is merely about reactions. Well I reckon that reactions can tell you a lot. But, as Godfrey rightly says, you have to have a well crafted questionnaire and far too many fall well short. As a final thought, getting good reactions doesn't guarantee good learning or business benefits. But getting bad reactions does pretty much guarantee lower levels of learning leading to fewer organisational benefits. Graham Graham O'Connell
Evaluating to level one is not simply a question of handing out a set of smile sheets, Godfrey Parkin offers some alternatives.
