An interview with Donald Kirkpatrick: The father of evaluation
Like many of the enduring 'greats' Donald Kirkpatrick has withstood the test of time and refined, tuned and tweaked a model of evaluation that he first published back in 1959. It's a theory he has never strayed from and loyal to this day it has been his life, his work and his love. Annie Hayes talks to Kirkpatrick about the model that has defined his very existence.
His story begins back at the University of Wisconsin: "I was teaching and I thought to myself 'as long as I'm running seminars why don't I evaluate them'," says Kirkpatrick who humbly admits that all he really wanted to do, to begin with, was to evaluate the reaction. "I found though this wasn't enough, I wanted to know if they'd learned anything but then that didn't seem enough either. I needed to know if they'd taken anything back to their jobs, whether they changed their behaviour and got positive results," explains Kirkpatrick.
The seed of thought formed the basis of his now famous evaluation model based on four tiers:
Reaction: what they thought and felt about the training
Learning: the resulting increase in knowledge, skills and/or change in attitudes
Behaviour: the extent of behaviour change
Results: the impact on the business resulting from the learner's performance
"I was teaching and I thought to myself 'as long as I'm running seminars why don't I evaluate them'. I found though this wasn't enough, I wanted to know if they'd learned anything but then that didn't seem enough either. I needed to know if they'd taken anything back to their jobs, whether they changed their behaviour and got positive results."
"I got my PhD and five years later, the editor of the American Society for Training and Development (ASTD) asked me if I wanted to write an article based on the evaluation model. I'd forgotten about it by then," he laughs, "But I said sure and I wrote four articles, one on each level," says Kirkpatrick.
Still hanging on his words It was a move that paid dividends – 40 years later and trainers and evaluators are still hanging on Kirkpatrick's words. In 1993 he published a book in which he profiled case studies to enable the readers to borrow concepts, forms and procedures as they wish. Dave Basarab provided one such case study showing how they had implemented the four levels at Motorola. He also wrote an introduction to the book which was called, 'Evaluating Training Programs: The Four Levels'. Book followed book and then Kirkpatrick's son, Jim, picked up the gauntlet with his own title, 'Transferring Learning To Behaviour'.
Simplicity is the key So why does this model still hold currency? I asked Kirkpatrick, who told me the formula is very basic: "It's practical and simple." With that simplicity is the ability to translate it into organisations of different structures, languages and industries – it can be drip fed out whatever the culture or dynamics: "Every company has a concern about whether they are getting their money's worth from the training programmes. Trainers can evaluate levels one and two, but they need the support of managers to evaluate levels three and four," explains Kirkpatrick.
Over the years Kirkpatrick has witnessed a step change in the way organisations evaluate their own training: "When I wrote my first book most organisations were measuring reaction but very few were measuring learning. Over a period of time, more and more top executives - the jury that would approve the budget! - were asking for more evidence than just the number of training programmes that were offered and the reaction to those programmes. And more and more organisations were putting more pressure on the training department to justify their budget."
Since the original model was published Kirkpatrick has made some changes: "We decided that 'reaction sheets' weren't good enough, now we advocate getting people together in focus groups, give them reaction sheets and follow up on it." Kirkpatrick is also a stickler for staying with the rules: "No it's not good enough to skip level three, for example."
A legend and a guru You get the impression that Kirkpatrick is amazed at the success that he is much trumpeted for. When the ASTD first called him a 'legend' and a 'guru' he laughed: "All I wanted to do was get my PhD done," he chuckles. It was a stroke of luck which he refers to as a 'miracle' and it has dominated his whole life: "Now practically everything I do is on evaluation. I have written a book on coaching but time after time they [organisations and conference organisers] still want me to do the same session on 'Evaluating Training Programmes: The Four Levels' because there are always new trainers attending the conferences and those who heard him before wanted to hear it again and pick up some practical ideas on levels three and four which they had missed the first time," says Kirkpatrick.
"When I wrote my first book most organisations were measuring reaction but very few were measuring learning. Over a period of time, more and more top executives - the jury that would approve the budget! - were asking for more evidence than just the number of training programmes that were offered and the reaction to those programmes. And more and more organisations were putting more pressure on the training department to justify their budget."
A sense of humour... Kirkpatrick is ever the practical man and he shares a story that conveys the good humour and ability to keep grounded that has kept him at the top: "I was conducting a session on 'Managing Change' at Dana Corporation in Michigan City, Indiana. Bill, the coordinator called me and asked if I was going to use the SMART approach. I wasn't quite sure what it stood for so I didn't want to say yes or no, so I humbly asked him to tell me what it stood for. He made the remark, 'I am surprised you didn't know'. So I told him I was going to use the PIE approach (I made it up just then and wanted to embarrass him a little.) After a pause he said 'I beg your pardon?' So I repeated it. And he said, 'Did you say the PIE approach?' So I said, 'I am surprised you aren't familiar with it.' So I explained that it stood for Practical, Interesting and Enjoyable. Both my son Jim and I use it today, except we have changed 'Interesting' to 'Interactive'," says Kirkpatrick.
But serious application Another side shows Kirkpatrick to be very serious about the application of his model. As the father of evaluation, he sends the following stark warning to training departments about loitering around levels one and two. "If your jury is not asking for more evidence now, they will be and you better be ready with some evidence from 'evels three and four."
Whilst the world has changed around him, Kirkpatrick's model has remained solid, simple and easily digestible – a formula that has been translated across the world into Spanish, Polish and Chinese to name but a few. Despite its longevity the word continues to spread and will do so for the foreseeable future.
Donald Kirkpatrick has very kindly agreed to answer any of your questions, so please take up this opportunity to ask about the secrets of evaluation. You can either post a comment at the end of the feature, or email features@trainingzone.co.uk We'll publish the answers at a later date. The deadline for submissions is the end of next week – 14 November – to give you all a chance to compile something. This really is a rare and golden opportunity to get advice from a master.
Annie Hayes, MCIPD, is a former editor of our sister site HRZone.co.uk. She now works part-time as contributing editor for both TrainingZone.co.uk and HRZone.co.uk using her wealth of knowledge to write features for both websites
TrainingZONE 10-Nov-08 Categories: Story read 5507
Allie Noel , 07 November 2008 @ 11:01 AM Making it Happen
Hi Don Most training professions in the public sector departments I work with say that they would like to evaluate at Levels 2, 3 and 4 and ROI, but lack the skills and resources within the department. For example within a hospital situation, given the demands of keeping day to services going, the time and money required to embed in depth training evaluation is offputting to HRD teams. Where they don't have cost or profit centres they also sometimes don't want to draw attention to the costs of training.
How would you go about persuading colleagues in this situation?
Many thanks and best wishes Allie
Tery Tennant , 06 November 2008 @ 17:13 PM Sustaining the Process
Jo-Anne:
I hope you don't mind my feedback here. I certainly also look forward to hearing Mr. Kirkpatrick's.
If good baseline measurements are set up front, then it becomes a fairly simple matter of tracking the changes, and keeping them alive. Let's say there is a particular action or behavior that needs to be adopted by a participant. By determining how this should be measured, this metric then can be used on an ongoing basis. It can be included in things like performance reviews, wall charts, review meetings, etc.
The key is establishing with the participant what the purpose is and what the follow-up schedule will look like. Then, whatever system is used, everyone involved will know the progress. Management should do its best to be positive and supportive of the changes, and be there to remove obstacles, etc. (This includes cultural obstacles.)
Jo-Anne Phillips , 06 November 2008 @ 12:24 PM Introducing a broader programme for evaluation
I have been working on introducing a broader process of evaluation and have secured up to level 2 of the process. I am currently working on the implementation of level 3 and would really benefit from some advice on how I might make the transition successful, and as with Bhavna's comments, what is the secret of sustaining the process?
Kon Stoilas , 05 November 2008 @ 21:41 PM The Don is the man!
I love the simplicity of this model. I also consider the following: Evaluate both the results and processes and learn from both success and failure. It is important to review with the perspective of contribution to customers' satisfaction, overall success of the company, and our own personal development. We need to learn from both successes and failures, and accumulate it as knowledge and know-how. So, in summary: (1) Evaluate the results and the processes and share it with members involved (2) Evaluate from key perspectives: Customers', The Business, and Your Own (3) Understand the reasons of success and failure. Kon Stoilas
Thanks so much for your simple but effective (and enduring) contribution to focusing on L&D outcomes! Since our business is all about pushing quantifiable goal achievement and behavior change, we are constantly talking about the four levels.
A while back, Jack Phillips postulated there was a level 5 to be added, which addressed ROI. I have maintained that there isn't really another level, and his idea is simply a measurement approach to the existing four.
I believe a better approach is implementing a baseline measurement, before L&D activities take place, and that this will generally facilitate the needed effectiveness measurement. This seems to inherently fit with your model.
Bhavna Patel , 05 November 2008 @ 13:50 PM Making the Evaluation process sustainable
I have used your model in different companies, and what I have found challenging is making the process sustainable, with constant changes in not only management teams but also in HRD function. What would you thoughts be on making the evaluation process sustainable? Bhavna Patel
Leanne Hoagland Smith , 05 November 2008 @ 13:24 PM Evaluation & Public Education
Don, As a proud owner of one of your autographed books, I have two questions. First, if you could share why public education in America (k-16) continues to ignore your 4 Levels of Evaluation. Given that most evaluation is at Level 1 and 2, when Level 3 is initiated such as through some state to national legislation, why do educators fear the accountability embedded within Levels 3 and Level 4.
The second question has probably been asked before, but please help me understand why so many HR professionals to consultants still insist that you cannot track Level 4 in quantifiable terms? In other words, why do they fear Level 3 and even more so Level 4? Leanne Hoagland Smith
Like many of the enduring 'greats' Donald Kirkpatrick has withstood the test of time and refined, tuned and tweaked a model of evaluation that he first published back in 1959. It's a theory he has never strayed from and loyal to this day it has been his life, his work and his love. Annie Hayes talks to Kirkpatrick about the model that has defined his very existence.
