Struggling with your ROI figures? Godfrey Parkin suggests you put your pen down and try a different form of evaluation.
There is a common misconception in business that, because they work with them all the time, financial people understand numbers. They like to reduce everything to money – what did it cost or what did it make? They insist on dealing in certainties and absolutes, where every column balances to the penny. But the real world does not work like that. The real world is characterised by imperfections, probabilities, and approximations. It runs on inference, deduction, and implication, not on absolute irrefutable hard-wiring. Yet we are constantly asked to measure and report on this fuzzy multi-dimensional world we live in as if it were a cartoon or comic book, reducing all of its complexity and ambiguity to hard financial “data.”
We struggle for hours (often for days or weeks) to come up with the recipe for “learning ROI.” The formula itself is simple, but the machinations by which we adjust and tweak the data that go into that formula are anything but simple. Putting a monetary value on training’s impact on business is fraught with estimation, negotiation, and assumption – and putting a monetary value on the cost of learning is often even less precise. Yet when was the last time you saw an ROI figure presented as anything other than an unqualified absolute? If you tried for statistical accuracy and said something like, “this project will produce 90% of the desired ROI, 95% of the time with a 4% error margin,” you’d be thrown out of the boardroom. You simply can’t use real statistics on an accountant, because the average bean-counter can’t tell a Kolmogorov-Smirnov from an Absolut-on-the-rocks. Don’t tell us the truth, just give us numbers that conform to our unrealistic way of measuring the business.
We spend way too much time trying to placate financial people by contorting our world to fit their frame of reference, and we allow them to judge and often condemn our endeavours according to criteria that are irrelevant or inappropriate. Perhaps there is some comfort in knowing that the problem is not unique to training. In my years in marketing, I saw plenty of good brands ruined by ill-conceived financial policies, usually to the long-term detriment of the company as a whole.
But you don’t need to be a statistician or an accountant to make a strong business case based on logic and deduction, and there is no need to be pressured into using the preferred descriptive framework of a book-keeper. The pursuit of the measurement of ROI in training is often a red herring that distracts from the qualitative impacts that our work has on the performance of the business. ROI is typically not the best measure of that, and, after making all of the heroic assumptions and allocations needed to arrive at it, that magic ROI figure may well be a false indicator of impact. Unfortunately, the indicators that are useful and reasonably accurate are often hard to convert to financial data, so they do not get taken seriously. And, compounding the problem, training managers themselves often ignore these indicators because they are not captured at the course level. Our focus too often is on the quality of courses rather than on the quality of our contribution to the business in total.
We need to widen the focus. While learner satisfaction, test results, and average cost of bums-on-seats are useful metrics, it is only after our learners have returned to work that we can begin to see how effective the learning experience really was. What are some of the indicators that let us know how we are doing? Many of them are produced already, often by the financial people themselves, and tracking them over time gives good insights into where we are doing well and where we might need to pay more attention.
Some of those metrics include: * Training costs per employee. * Enrolment rates and attendance rates. * Delivery modes, plans against actuals. * Percentage of target group that is “compliant”. * Time from eligibility to compliance, or to proficiency. * Percentage of workforce trained in particular skill areas. * Learning time as percentage of job tenure. * Availability, penetration, and usage rates of help systems. * Skill gap analyses tracked over time. * Productivity (such as, for example, number of new clients per 100 pitches). * Attrition rates.
There are many, many more. Metrics such as these let us put on the manager’s dashboard indicators of performance in areas such as operational performance, compliance, efficiency, effectiveness, and workforce proficiency, as well as harder to capture dimensions such as motivation and readiness for change. Training departments need to think “outside the course” and come up with ways to derive the right indicators in a way that is inexpensive and unobtrusive.
One of my favourite recommendations is that training departments learn something from their marketing colleagues, and set up the ability to run surveys and focus groups, to investigate learner satisfaction, customer attitudes, job impacts, attitudes, and manager perceptions. This skill is often absent in training departments, which is a pity because these methods can produce great insights and save money and time. If you build this capacity into your training organization, getting a read on Levels 3 and 4 can become as much a part of your evaluation regimen as gathering smile sheets. You don’t have to interrogate the universe if you can pick a small sample. And you can produce real data and real trends that go down very well in the boardroom.
* Read more of Godfrey Parkin's columns at Parkin Space.
Paul Wadsworth , 27-Jul-05 Training Return - actual or coincidence?
Some years back I pursuaded the company I worked for then to run Team Building Training for all Factory staff - about 160 people total. This was delivered as a 2 day, off-job session, by an external provider - to give a measure of 'independence' to the training.
At around the same time the company spent £100K's on upgrading a 24/7 production line in the expectation of, at best, achieving +25% in output. When the upgrade had settled the improvement was ~33%.
I pointed out that if 1 percentage point of the over-target improvement was due to the Team Building training the £'s gained from this extra 1 percentage point of output would more than pay for the Team Building training - both the Trainer's fees and the delegates hours - for the entire programme!
Could this be 'proved'? Of course not - after all we didn't do a 'control' whereby 2 from the 4 shift-teams in that department were team trained and the other 2 shifts weren't.
But do you believe in 'coincidences' like this?? Paul Wadsworth
Joe Espana , 26-Jul-05 The purpose of ROI
As ever a very thought provoking article. But I don't necessarily agree with all its assertions. The thorny problem of evaluating the business impact of learning and development and its return on investment will remain as long as the function and its members continue to think about the focus of what it does in terms of inputs only, i.e the course, e-learning package etc.
One point that Godfrey makes which is a really valuable one is the one that suggests that training and development professionals have to apply skills beyond those normally associated with their profession. I would go further suggesting that skills, tools and approaches normally associated with marketing, valuable as they are. I would suggest that training & development professional have to really understand business management generally. The suggestion that was made over 20 years ago that HR / Training & Development have to speak the language of the business remains true today. Some of the difficulties of measureing learning ROI in terms of performance improvement is that the "owners" of that performance (line managers) themselves find it difficult to measure performance and contribution in terms of a value. because this is difficult at the front end, it then because vague and tenuous at the back end of any learning intervention when the poor Training Manager is trying to work on an ROI evaluation project. I suggest that it is partly the responsibility of learning and development professionals to 'educate' coach and support line managers to determine the value of existing performance, so that they can calculate what a % improvement would actually mean in business performance terms. This helps enormously in the process later when the real measures are being taken. As Paul Kearns and others have shown, added value on performance can only come from a very few number of variables. these need to be understoon by the learning and development professional as well as the line manager. The right questions need to be asked (usually several "whys")in order to help line managers understand the likely impact and the precision of the envisaged learning intervention. At the end of the day the only real purpose of ROI evaluation is to determine whether the intervention was worth doing in the first place. Teh answer to that can be determined, to a very great extent, before the intervention is designed and delivered.
One final point: If undertaking ROI evaluation is partly about demonstrating the valued contribution that the function brings to improved business and organisational performance,particularly when arguing over budgets, what I have found in corporate life is that it doesn't have to be undertaken that often. If ROI contribution is shown once (well done admittedly) the FD and other senior managers are more likely to listen and pay attention when discussing the value of the next intervention. Joe Espana
Geoff Coughlin , 25-Jul-05 Real-World Business Examples are very powerful...
Excellent piece Godfrey - I wholly agree with your thoughts. Many will know of the Fitzpatrick Model for evaluating the effectiveness of learning and development activities and this is a practical too, worth using. However, what I've found very powerful are good examples of where learning has been applied in the workplace to directly benefit the business, team and individual concerned. An example: About 2 years ago we were engaged to deliver a leadership and management development programme to a group of middle and senior managers operating in a computer retailing environment. I had completed delivering the module focusing on communication skills and 3 weeks later the Sales Manager (a participant) called me to say how useful the session in question had been. He had been tasked by the CEO to get on the phone to an educational establishment and "get that cheque for £1.5 millions - now!". This was for the supply of computer gear and they weren't paying up because of some service issues. The Sales Manager said - "You know Geoff, if I had approached my contact the way I usually do, we'd have got nowhere...definitely would have ended up shouting at each other - this time I used some of the assertive principles we went through and I tried not to argue with him. Yes, they've still got some problems for us to sort out..but guess what? They've paid up - in two days...fantastic!". This is the kind of example that we can and perhaps should look for and report it back to those who need to hear - those accountants? The CEO? And many more. Thanks again for a good piece Godfrey. Geoff Coughlin
Doug Smith , 25-Jul-05 Quantitative AND Qualitative
Well said, Godfrey. The measures that you mention usually have far more significance (and accuracy) than contorted attempts to produce an enterprise-wide ROI.
While I do still believe that it's possible to quantify a training effort (though it takes the rigor of a six sigma statistician) there is as much, if not more substance in uncovering the qualitative benefits to each initiative. Impacts to employee satisfaction effect both customer satisfaction and retention and that can be captured with frequent surveys and candid conversations.
Thanks for stating a necessary truth.
Bob McKay , 25-Jul-05 Accountants "Lest we not forget"
When I was in corporate life, I often was engaged in the kind of debate you outline. Most of the time it was not so much to evaluate but to defend any spend on training and development. However the best line of attack to accountants was to remind them of the support they received both in paid time off and fees for them to complete each stage of their financial qualification.
To add insult to injury, it was the norm that when they successfully completed each stage they received a salary increase!
Struggling with your ROI figures? Godfrey Parkin suggests you put your pen down and try a different form of evaluation.
