Everyone wants an ROI study to prove the value of their training – or do they? Return on Investment is complicated and time consuming to prove and, argues, Donald H Taylor, usually only the training department is interested in the results.
Wouldn’t it be wonderful if we could prove training’s value in money terms?
Sure, really wonderful.
So why don’t we do it?
A lack of technique isn’t stopping us. Jack Phillip’s approach extends Donald Kirkpatrick’s 1959 four levels of evaluation and is a robust ROI method. There are other techniques, too. So why aren’t they more widely used?
There are two reasons.
First, a practical reason: the ROI of training is both time consuming and complicated to calculate. The second reason is a question of attitude and a fundamental show stopper. We’ll come to it later.
ROI basics First, the basics: what exactly is Return on Investment?
It is absolutely not a demonstration of cost saving, although many studies claiming to show ROI are nothing more than that – especially in the e-learning field. Rather, it is a way of showing what you get out for what you’ve put in.
Suppose you have installed some new equipment in a factory. The ROI calculation is clear: the plant cost you something, and your output increased after the installation. The ratio of the benefit to the cost is your ROI:
Return on Investment = incremental value / investment made
This should be expressed as a return over a period, and as a percentage. So if your new piece of plant cost £10,000, and over a year produces an extra £2,000 worth of value, it will yield a ROI of 20% per annum.
Splendid isolation People, though, aren’t machines, and making the same calculation for training proves more complex. Even for workers whose output can be clearly measured, such as some manual workers and sales people, proving that a change in that output was only the result of training, is complicated – there is just too much else going on to easily isolate the effects of the training.
If, after training, a group of employees shows an increase in productivity, is that down to the training? Not necessarily. We need to know that the change wouldn’t have happened anyway.
We need to know, for example, how that group performed before training. Suppose they are sales people – were their sales at a flat level, or already on an upward path? If the latter, then did the training result in an increase in sales beyond the trend that already existed?
And a historical trend is not the only way an increase in performance can be caused by something other than training. Perhaps the employer made internal systems more efficient, or a competitor went bankrupt, increasing demand. To see whether these exogenous factors caused any increase in productivity, we need to divide our sales force into two groups, with only one receiving training. This second group should be as similar as possible in every way to the trained group, from their age and educational background to their locations, experience and knowledge.
If we can set up a control group like this, and be sure what the historical productivity trend was, we can be fairly sure to identify any productivity increases stemming from training. I say fairly sure, because there is still the Hawthorne effect to consider, in which people increase productivity simple as a result of change/observation. (In the Western Electric Hawthorne electrical plant in the 1920s, workers’ productivity increased temporarily when factory floor lighting was improved, and then increased again when it was reduced.)
But if the Hawthorne effect can be accounted for (and it can), and if we have historical data and a control group, then ROI should be calculable (if we avoid for the moment the huge subject of calculating the monetary value of the output of non-sales staff).
Yet it was only after several years of false starts that Neil Rackham, a scientifically rigorous research psychologist and founder of sales training company Huthwaite was able to find an organisation willing to undergo a rigorous ROI trial – Motorola, Canada.
The results (published in Appendix A of Rackham's 1981 book SPIN Selling) did indeed show a significant increase in sales as a result of training. Even so, Rackham adds:
”There are even more tests I'd like to carry out before I'll be totally satisfied that the ideas I've described in this book will significantly improve the results of major sales.”
This man stood to gain considerably by proclaiming the validity of the training programme he had devised. If he felt less than totally convinced by his ROI study, this is surely a lesson for the rest of us. ROI is complicated; it’s very time consuming and it may not in the end be convincing to an expert in the field.
But there’s a more important reason why training ROI is just not worth the effort.
The wrong answer At the beginning of this piece, I said that there was a second, fundamental reason why people don’t carry out ROI studies on training. It is this:
Apart from training professionals, nobody really cares about training ROI.
If they did care, believe me, we’d be swamped with rigorous studies. The reason Rackham had trouble finding a company willing to be the guinea pig for his ROI study was not just the complexity of setting up a representative control group, or the political difficulty of explaining to half a sales force why they weren’t getting trained. It was the constant response he received: ‘Look we’re pretty sure that the training works well enough. Skip the ROI and just get on with it’.
In other words ROI is the wrong answer to the right question.
The question (coming from managers and directors) is: `How do I know this training is valuable to the organisation?' to which the answer from the training department need not be `because I can show you, through this exhaustive ROI study, the increased monetary value of productivity that will result'.
So what do they want if they don't want ROI? They want value. They want training (where training is the best solution) to make the organisation more effective. What exactly does that mean? I don't know, because I'm not in your business, but you should.
The value proposition Every organisation has operational pinch points. Not all of them can be improved by training, but where it will help there is usually a clear value proposition attached.
If there is a series of delayed project starts because of a lack of staff with key skills, targeted training can solve that. The value proposition: faster project starts. The time saved can be reckoned into a monetary value. That isn’t rigorous enough to be an ROI study, but it’s the sort of rough calculation that will justify (and well exceed) most training spend.
Is there a staff turnover problem with new recruits leaving too early? Improve and extend induction training solution for new joiners. Or better still, consider an electronic solution for briefing people before they join up. If they are going to leave anyway, train them in detail about what they will be expected to do, and have the eventual drop outs decide not to join – that’s cheaper for the organisation. Again, the cost of staff turnover far outweighs the cost of the training. The value proposition is clear and the CFO will love you.
Justifying on-going training for staff so that they can do their jobs is harder, because the benefits are further down the line. The best answer is to have already engaged in a series of exercises like those above that demonstrate value. More important than any ROI study, that will establish in the minds of the rest of the organisation that you understand the business issues, and are using that understanding in building your training programme.
Good and bad requests for ROI There are two reasons why someone asks for an ROI justification for training. Usually, they don’t understand the complexity of a real ROI study as described above. Put them right on that, and then discuss the business problem they’re trying to solve. That will root out the value proposition and get you talking on their wavelength.
The other reason for an ROI request is more sinister – they know exactly how difficult it is. Asking for a study ensures a long period of quiet while that pesky training person goes away and tries to sort it out. Don’t stand for it. Explain that a full study will need funding, and offer an alternative, value-based approach. Given the choice, anyone serious will opt for the latter.
A business value proposition that relates directly to a perceived organisational issue is where the training department can make a real, clear difference. Identify and solve these business issues and I guarantee that you will not be asked to provide an ROI analysis of what you are doing. Instead, you will be welcomed with open arms as an important part of the team.
About the author: Donald H Taylor is Alliances Director at InfoBasis, and Chairman of the Learning Technologies and IITT National Trainers conferences. In January he was presented with the Colin Corder Award for outstanding services to IT training. He has a regular blog at www.donaldhtaylor.co.uk and can be contacted at donaldt@infobasis.com. He has been a TrainingZone member since 2003.
Tery Tennant , 21-May-08 Liked the example Gary gave . . .
. . . regarding the NHS in his "May 1st" response (I don't know what's going on with the dates of these messages, as I just got e-mail notice of it being posed, and it also says "May 1, 2007"!). Very practical illustration. If we are purporting that some leadership development program is going to have good effect, than we should be prepared to match it up (ROI wise) with other organizational activities - including non-developmental things which have known ROI.
It seems ludicrous for someone like NHS, being awash in red ink, to be considering an expensive program that they can't even project the return on - makes little business sense! (but then again, that's government for you - it's only tax revenue, which they don't really much compete for)
Yes - Merry 'what ever' you celebrate at this time and a good new year to you. Garry Platt
Donald H Taylor , 24-Dec-07 Merry Christmas
Well, Garry, I think that's a wrap for this year.
I'm sure that after all these months, nobody else is reading this, and feel that the dialogue has probably reached a natural end, so let me take the opportunity to wish you a Merry Christmas, and all the best for 2008.
Donald, I think you can call it what ever you like. I suspect that many others having read your article and then contrasting it against the criticisms outlined here:
will realise that your opinion; that it is virtually impossible to undertake an ROI successfully is actually just obtuse thinking and extreme analysis. Few people I suspect will share your position who wish to adopt an ROI approach and also use reasonable practises.
Garry, I think we’re going around in circles. As I said in my last post: Some measure is possible. I just wouldn't call it ROI.
You obviously feel very strongly about this, but I believe that we have reached a position with no resolution, where everything depends on the meaning of words.
Donald Taylor wrote: “I think you may be missing the point of the article. You write: What you described as a process of ROI is not one I recognise as being either realistic or workable. I agree. It isn't workable.”
I do not think I am missing the point, you’re just ignoring it! The *point* is Donald that the bizarre methods you described are NOT the only approach to ROI in training that exists and you seem to avoid that issue. The reality is that you CAN undertake an ROI of training without using the system you outline.
Donald Taylor continued: “And that a ROI of training is occasionally necessary, feasible and possible without the degree of puerile decontamination and secondary analysis you described. I agree. Some measure is possible. I just wouldn't call it ROI.”
Return on investment is an analysis of how much money accrues to the host organisation over say a period of two years as a result of the training. The base line method can achieve this without the nonsense defined in the article. The fact that you don’t recognise this or feel that so many contaminants muddy the picture rendering the results useless, but exactly the same variables can impact on countless other financial analysis but the results are accepted as valid and reasonable seems a double standard and as I said untenable.
I think you may be missing the point of the article.
You write:
What you described as a process of ROI is not one I recognise as being either realistic or workable.
I agree. It isn't workable.
You go on:
And that a ROI of training is occasionally necessary, feasible and possible without the degree of puerile decontamination and secondary analysis you described.
I agree. Some measure is possible. I just wouldn't call it ROI.
Donald Taylor wrote: “It comes down to this: a rigorous ROI study needs what I describe above.”
Bluntly; no it does not. This definition and the approach you have described here of ROI is an exercise for pedants and does a disservice to the process of ROI. It is not a realistic approach and is more akin to some academic research study not an everyday financial breakdown.
Donald Taylor continued: “Without it, any study becomes something else - perhaps an operation in demonstrating value. That's fine, and it can be useful. It just isn't an ROI.
This would be true Donald if we all applied *your* definitions of how ROI should be undertaken, and how all possible contaminants like the Hawthorne effect should be eliminated, a notion in my opinion that goes beyond the realms of common sense in this context. The points you raise are I think an argument of convenience and not one based in actuality; they are merely fractious criticisms to support your views.
Your position in my opinion is untenable when you compare this approach to other business analysis undertaken in the workplace. Influencing factors are recognised as potential players but are not tracked to fifteen decimal places, nor do they in other areas of business, marketing, sales, absence management, quality control etc etc. With your approach to ROI there appears a morass of internal and external variables which have to be considered like the process is some kind of quantum physics equation involving dark matter and sub atomic particles. I chose to work in much more realistic way as do many of my colleagues.
Donald Taylor finished with: “So, it seems we both agree on two things: that a study to the degree of detail I describe above is usually pointless, and that it is nonetheless important to demonstrate the value of training.”
Your assumption that we agree on this issue is not correct. What you described as a process of ROI is not one I recognise as being either realistic or workable. And that a ROI of training is occasionally necessary, feasible and possible without the degree of puerile decontamination and secondary analysis you described.
Donald H Taylor , 17-Dec-07 Value is the crucial matter
Thanks, Garry.
It comes down to this: a rigorous ROI study needs what I describe above. Without it, any study becomes something else – perhaps an operation in demonstrating value. That’s fine, and it can be useful. It just isn’t an ROI.
So, it seems we both agree on two things: that a study to the degree of detail I describe above is usually pointless, and that it is nonetheless important to demonstrate the value of training. All we disagree on is the name to give that demonstration of value.
Donald Taylor Wrote: “The argument in this piece is that following the mathematical definition of ROI leads to a particular method of implementation which is time consuming and difficult.”
I think this was clearly refuted on two counts here:
1. The model you were focussing on for undertaking ROI analysis is probably one of the most difficult and time consuming models of the lot.
2: The level of analysis and the unnecessary safe guards against contamination were innapropriate when using a modicum of common sense.
Donald Taylor continued: “Also, it is not something organisational managers are usually looking for.”
I wonder however how many managers know that it is possible to do and also that it is not ‘time consuming and difficult’?
Don Taylor added: “They are looking, instead, for evidence that learning interventions have contributed towards organisational goals. That evidence can – and should – be provided in other ways.”
Yes it can, in many cases the ROI approach is pointless and worthless, but it is not always the case as you seem to be suggesting here, and there are many cases when an ROI analysis is essential if not imperative. And since when has the CIPD been at the cutting edge of training?
Donald H Taylor , 17-Dec-07 Reflecting and looking forward
As this was one of the most read articles on Training Zone in 2007, I thought it was time to reflect on it.
The argument in this piece is that following the mathematical definition of ROI leads to a particular method of implementation which is time consuming and difficult. Also, it is not something organisational managers are usually looking for.
They are looking, instead, for evidence that learning interventions have contributed towards organisational goals. That evidence can – and should – be provided in other ways.
This issue of showing the value of training and learning will undoubtedly become increasingly important in 2008 – what else does the L&D function exist for?
Garry Platt , 13-May-07 Jon Ingham - Some responses:
Jon Ingham wrote: “The recent survey from the CIPD and last year’s research by IBM / ASTD show that calculating ROI provides very little value in many cases in most organisations. It’s something L&D often thinks it needs to do, but actually most business leaders just aren’t interested.”
I disagree, if we are referencing the same CIPD survey, it stated: “Based on current academic and practitioner work and also the Value of Learning discussion thread results we produced a poll investigating the barriers to evaluating learning and invited members of the network to participate online. In all 392 CIPD members contributed during the period November 2006 to January 2007.” The results stated that for 70% of the population ‘Serious evaluation is too time consuming/costly and we have other priorities’. So the larger part of L&D specialists (based on this analysis) do not appear to think they need to undertake ROI analysis – and in my opinion many Business Leaders aren’t interested in the ROI process because they have been fed the line ‘it’s impossible or pointless’, which some L&D practitioners believe.
Jon Ingham wrote: “However, the debate hasn’t yet dealt with what is the main issue for me. This is intangibility. The greatest value from most transformational development comes in the form of intangibles, which can’t be attributed to direct financial benefits, and therefore can’t easily be included in a ROI calculation. This is intangibility.
The greatest value from transformational development might in your experience come from ‘intangibles’ but not in mine. I would suggest that if you are undertaking training based upon an Identification of Training Need (ITN) which has only produced a list of ‘intangibles’ that it will output you haven’t undertaken an ITN that I would deem adequate and/or you are simple undertaking training which probably does not have a sound business case to support it and will be of no net benefit to the host organisation.
Perhaps Jon could provide an example of how "The only meaningful way to provide a ROI on intangibles is to do this in qualitative terms" Paul Kearns
Jon Ingham , 08-May-07 ROI should be a conversation
I think Donald Taylor provided a very soundly argued case for considering other forms of evaluation to ROI (absolutely not a diatribe!). The article was certainly challenging but I think that this is what is needed to jolt people out of the common perspective based on rather lazy thinking that ROI is the holy grail of learning evaluation.
It’s interesting that neither of Donald Taylor nor Gary Platt, nor many of the people who have added comments to these recent debates, seem to be suggesting that ROI is a holy grail. I’m pleased with this, but it doesn’t detract from the fact that many, many senior L&D practitioners who should know better, consider it as such.
The recent survey from the CIPD and last year’s research by IBM / ASTD show that calculating ROI provides very little value in many cases in most organisations. It’s something L&D often thinks it needs to do, but actually most business leaders just aren’t interested.
For what it’s worth, my own perspective on the debate so far is that it would only very rarely be worthwhile making the investment to implement Jack Phillip’s methodology, and that Paul Kearn’s approach would only ever be suitable for small, transactional pieces of training (it not being possible to assume that nothing else will change – the condition that goes along with not using control groups – for more significant, longer-term development).
However, the debate hasn’t yet dealt with what is the main issue for me. This is intangibility. The greatest value from most transformational development comes in the form of intangibles, which can’t be attributed to direct financial benefits, and therefore can’t easily be included in a ROI calculation. Paul Kearns ignores these intangibles unless they’ve got a pound sign attached (which is impossible to do when the output of the learning and its impact on the business are distant in time and space), and Jack Philips leaves them aside until after the ROI has been calculated without them (which is clearly nonsensical).
The only meaningful way to provide a ROI on intangibles is to do this in qualitative terms. This isn’t making the best of a bad job, it’s using an appropriate measurement framework to deal with the type of value that’s being measured.
Training ROI should be the conclusion from a conversation, not a number.
For a detailed response to this article. Garry Platt
Garry Platt , 01-May-07 ROI
Mike Taylor wrote: "The other is that the costs of "measuring ROI" outweigh the benefits."
This can indeed be true. There are many occasions when undertaking an ROI would be an utter and complete waste of time, e.g. when the training is compulsory and regardless of the outcome it would have to be continued anyway. It can also relate to events which the host organisation recognises as loss making but because of contextual and mitigating circumstances is prepared to absorb that cost. There are also other situations where ROI however is not only relevant but essential. In these circumstances I would question the validity Mike Taylor’s assertion.
Let me give an example where I suspect a Base Line i.e. Kearn’s Model ROI analysis would reveal exactly where the training is working and not working regardless of what the managers believed. (I would not be employing the Jack Philip’s approach as I believe it to be to inefficient and expensive to undertake.) In June of 2005 the Modernisation Agency within the National Health Service published a guidance paper on the evaluation of Leadership Development within the NHS.
This is a 68 page report which outlines an excellent range of suggestions and ideas for undertaking evaluation at levels 1-3 of the Kirkpatrick model. In relation to ROI it contains less than 100 words and essentially dismisses the approach. This is an organisation that at the end of this year will be £500M in deficit and is declining to offer life saving and life extending treatment on the grounds of cost.
Within this context I would be looking at major management development initiatives which are being rolled out within it and focus on the intended outputs these programmes are aimed at achieving. Is it delivering increased efficiencies and improving the effectiveness of the NHS? How much is it costing and is it a responsible investment? A new range of MRI machines are currently being used in Wythenshawe Hospital, Manchester area due to their increased picture resolution (proven), their operational reliability (proven) and the lower running costs (proven). It’s a sound investment which has been verified and makes sense to continue and extend elsewhere. Where significant amounts of money are being diverted from hospital budgets towards Team Building Events and Leadership Development Programmes I would hope the same sort of rigour should be applied. It isn’t only money in this context, it’s peoples lives and jobs.
Finally let me reiterate, ROI is not essential or even important for every form of training, but in some cases it can be an important contributor to making sound decisions to extend, change or finish the training. A total denial of the usefulness and relevance of ROI in many circumstances doesn’t make any sense.
Mike Taylor wrote: "The other is that the costs of "measuring ROI" outweigh the benefits."
This can indeed be true. There are many occasions when undertaking an ROI would be an utter and complete waste of time, e.g. when the training is compulsory and regardless of the outcome it would have to be continued anyway. It can also relate to events which the host organisation recognises as loss making but because of contextual and mitigating circumstances is prepared to absorb that cost. There are also other situations where ROI however is not only relevant but essential. In these circumstances I would question the validity Mike Taylor’s assertion.
Let me give an example where I suspect a Base Line i.e. Kearn’s Model ROI analysis would reveal exactly where the training is working and not working regardless of what the managers believed. (I would not be employing the Jack Philip’s approach as I believe it to be to inefficient and expensive to undertake.) In June of 2005 the Modernisation Agency within the National Health Service published a guidance paper on the evaluation of Leadership Development within the NHS.
This is a 68 page report which outlines an excellent range of suggestions and ideas for undertaking evaluation at levels 1to 3 of the Kirkpatrick model. In relation to ROI it contains less than 100 words and essentially dismisses the approach. This is an organisation that at the end of this year will be £500M in deficit and is declining to offer life saving and life extending treatment on the grounds of cost.
Within this context I would be looking at major management development initiatives which are being rolled out within it and focus on the intended outputs these programmes are aimed at achieving. Is it delivering increased efficiencies and improving the effectiveness of the NHS? How much is it costing and is it a responsible investment? A new range of MRI machines are currently being used in Wythenshawe Hospital, Manchester area due to their increased picture resolution (proven), their operational reliability (proven) and the lower running costs (proven). It’s a sound investment which has been verified and makes sense to continue and extend elsewhere. Where significant amounts of money are being diverted from hospital budgets towards Team Building Events and Leadership Development Programmes I would hope the same sort of rigour should be applied. It isn’t only money in this context, it’s peoples lives and jobs.
Finally let me reiterate, ROI is not essential or even important for every form of training, but in some cases it can be an important contributor to making sound decisions to extend, change or finish the training. A blanket denial of the usefulness and relevance of ROI in many circumstances doesn’t make any sense.
One theme that seems to run through this is that good managers (of which there are many) should and do know what is required, and whether its working. It's what they are there for.
The other is that the costs of "measuring ROI" outweigh the benefits. Mike Taylor
Donald H Taylor , 27-Apr-07 Wrap up for this week
Thanks for all comments. I welcome all continued, civilized discussion.
To summarize: we all agree that some measure of value is crucial for training. My point is that the measure used needn’t always be the rigorous proof of an ROI study. Demonstrate that a training intervention meets a particular business need is often enough.
Given that, here are some replies:
Martyn – thanks for your level-headed contribution. I would encourage readers to read the report you cite and to join in the CIPD’s debate.
Garry - I had read your piece on the Holy Grail of ROI before writing this article, and agreed with it, so I am a little surprised at your reaction. My article is not meant as a diatribe, rather I aim to suggest that ROI is a term more abused than understood, and that other less stringent methods of judging the value of training are available and usually applicable.
James – you’re quite right. Large investments running into millions of spend should indeed be backed up by a sound ROI argument. I should have been clearer that my argument does not apply in this case, but was rather aimed at the daily world of training, where the term ROI is bandied about as if it were trivial.
Paul – we are clearly agreed that you can show the business value of training in most cases (but see James’ caveat) without having to use trend data and control groups. You have your own approach, which looks to have been successful.
Dominic – I look forward to reading your success stories in the near future.
Meir – apologies for not responding in public to your comment. As I noted in our e-mail correspondence, I agree that customer satisfaction is generally a good practical gauge of value for most training interventions, if the buyer knows how the training is aligned to their business need. If not, there is a danger of what Charles Jennings of Reuters calls the ‘conspiracy of convenience’, where the training function provides what the manager asks for, but neither is actually addressing the business need.
Tery Tennant , 27-Apr-07 I look forward to Gary's article . . .
and maybe Paul Kearn's could be persuaded to do one too!
As he advocates, we are finding the use of baselines to show value (whether it be strictly monetary or not), to be simple & effective. None of our smaller business clients are pushing to have thorough ROI studies performed. They want to see demonstrated value, and that can usually be done to their complete satisfaction if baselines/measurements are established, and then reporting on the specific progress. It keeps things focused & aligned with what matters most.
Again, why use a 5 pound sledge hammer on a gnat? While my analytical side would actually love doing this, most of our clients simply wouldn't tolerate it! Tery Tennant
Garry Platt , 27-Apr-07 Response Article Shortly
Within the limited space one is afforded in this response section I cannot begin to detail some of the errors, omissions and misconceptions this article perpetrates and sustains. Consequently I have approached Claire Savage to see if she will accept a ‘Response Article’ similar to the one I wrote concerning ‘The Holy Grail of Evaluation’, see:
Fortunately she has agreed. This should have a sufficient word count to be able to present an alternative view of the role of ROI and how it works which is based upon a practising and informed view which I feel is somewhat missing in this diatribe.
Thanks, Donald. I also sense that we're closer to agreement than our initial comments might have suggested. I think that I was provoked by such statements as "Why Training ROI Doesn’t Matter" (your title) and "training ROI is just not worth the effort"
I would be delighted to share my experiences with Claire - her name rings a bell. I also agree that applying ROI methodologies is often tricky - and that a reliable method of isolation (I tend to use control groups too - can't get my head around the others which JP describes) and good trend data are essential in this regard.
I'm also not sure that the Hawthorne effect (which must apply) necessarily invalidates the attribution to the intervention. On the contrary, if some or all of the effect is down to the fact that what is watched/measured improves then those who include "watching/measurement" as a vital component of their intervention should still be credited as having facilitated the resulting impact. What takes place in a training room is no magic wand. Hopefully, we all realise by now that it's the attention that something is given in someone's day to day environment that contributes most significantly to performance-improvement. Dominic Johnson
James Graham , 26-Apr-07 Simplistic & lightweight
I found this article very disappointing.
If the author was trying to start a debate by reaction to a controversial generalisation, perhaps he succeeded.
However, I found that he tried generalise where my experience suggests that there is the need for a great deal of particularisation.
Some training projects are of an order of magnitude and complexity that a value proposition approach makes sense, but others are not.
To be taken seriously by a hard nosed board when asking for sign off of many millions often requires ROI to be provided.
Anyone who thinks this is never the case is a fool and does not advance the L&D profession.
Tery Tennant , 26-Apr-07 Baseline is key
I certainly concur with Paul about baselines. When we started focusing more on getting the baseline (exactly what is happening now), a number of good things came out. Clients more quickly got on board with seeing & measuring their own ongoing progress. Before you can figure out where you have gotten to, you've first got to know where you started. This has to happen prior to the development actions.
We have found this simple realization then drives the process quite effectively. Tery Tennant
Martyn Sloman , 26-Apr-07 New research report
Those who are following the debate may be interested to know that the CIPD have just published a Change Agenda: the Value of Learning: a new model of value and evaluation. This is based on research undertaken by the University of Portsmouth in 13 organisations and suggests that many different methods of evaluating learning are now advocated by consultants, academics and commentators. These include two sorts of measures. First, those that use ‘hard’, numerical data about training that has already taken place (for example return on investment measures; learning function efficiency measures); and secondly, measures that use both ‘hard’ and ‘soft’ (qualitative) data to assess the present and future contribution of learning (for example return on expectation measures; scorecard and capacity measures).
Getting the baseline sorted out at the beginning makes your questions unnecessary.
But for a fuller (free)answer see my TJ series at www.paulkearns.co.uk - what is interesting is that in those 12 articles ROI did not feature very much and did not warrant a whole article to itself - it is, as I said before, just one tool.
I think an even more ironic question is why does it take so long for the 'learning profession' to learn?
There seems to be a lot of confusion here and most of it stems from common misunderstandings about ROI. 1. Jack Phillips does not resolve the ROI issues before the training starts - that's why his methodology has to 'convert' to $'s 2. ROI studies are not complex as long as they are well designed at the beginning. 3. Isolating variables(again a notion reinforced by Phillips)is a red herring. Similarly, there is no need to conduct control group experiments - no wonder you think ROI is a chore Donald.
As I said below - ROI can be a very powerful tool - but only in skilled hands. Trainers who find it difficult are usually using the wrong models - that's why I spend a great deal of my time teaching them how to keep it simple and avoid these pitfalls perpetuated by theories that are not robust. Paul Kearns
Donald H Taylor , 26-Apr-07 Let's keep talking
Thanks for all these comments. It seems that we’re all agreed: some measure of value is crucial for training.
My point in this article is that the measure needn’t be ROI. As many of the comments point out, a demonstration of alignment with the business and performance improvement after training is often enough, and usually more effective than a rigorous ROI study.
While the formula for ROI is not complex, carrying out a true training ROI study undoubtedly is. To reiterate: it requires isolating the effects of training by using both trend data and a representative control group. Then a monetary value needs to be assigned to the effects of any improved performance. That is easy to describe in theory, difficult to do in practice.
Of course such studies are possible, but running them effectively is the trick. Dominic, it sounds like you have some good stories to share. I have spoken to Claire Savage this morning, and I’m sure she’ll be in touch – a strong proof of training value always deserves a wide hearing.
I sense one area where I may not have been clear in this article. I am not attacking Jack Phillips and his methodology, else I would not have asked him to keynote at the Learning Technologies Conference 2006. I still believe his approach to be robust, and accept Jack’s point that it is not always applicable.
I am delighted at the scope and thoughtful nature of these comments. Training's value is an important issue that deserves airing. Let the debate continue. Donald H Taylor
Peter Mayes , 26-Apr-07 Training is not alone.
Hi all
Talk about coincidences, was clearing away some old copies of Computing and in an issue from last July came across the following bulletin heading:
Firms fail to measure the benefits of IT. Some 60% of firms indicated that they found it difficult to demonstrate the value of IT.
Given the 'tangible' nature of IT (boxes in cupboards and on desks and so called time saving devices and applications), this would suggest that investment in IT is based on ... well … a wing and a prayer?
Training is a far more intangible activity and as a result more difficult to measure using clinical (accounting) analysis. The challenge then is to take a more balance approach. Now it’s a few years since I read ‘The HR Scorecard’ by Becker, Huselid and Ulrich but as a model it proposed some real challenges; challenges that would as suggested, separate out the “professional learning practitioner” from the “jobbing trainer”. I don’t profess to know whether this tool is good or not, but it adds to the portfolio available within the HR/training profession.
I had the pleasure of inviting Garry Platt from Woodland Grange to give the keynote session at the recent Trainers’ Conference run by TrainerBase. He commented that there is a professional and commercial gain available to learning practitioners who do or at least offer to undertake such analysis; irrespective of tool used, whilst acknowledging that there are good reasons for a ROI analysis not being conducted (when there isn’t a choice).
In (almost) all transactions there must be a perceived value to each element of (and party within) the exchange: I give you something of ‘value’ in exchange for something else of ‘value’. Take this a step further; shareholders of a company can challenge the expenditure on a purchase and if the purchase cannot be justified, hold the individual procurer to account: “And why did you spend £15,000 on this management course”? Response, “Cos I thought is would be a good idea”, or some other difficult to justify reason. This isn’t going to engender any level of trust in the judgement of that individual.
As a person who runs a business, albeit a micro business, value is a constant thought in my mind in all purchases: “what will I gain by spending this money, or time for that matter, that could be spent on something else”. The gain will not always be directly attributable to a financial benefit, but I will perceive a value based on other non financial gains, yes the soft stuff: emotional and spiritual benefits.
And how have I benefited from writing this response: its made me feel good, which has motivated me to do something, which means it will get done quicker and cost less: QED ???????
Peter AKA Ed Founder and Editor of TrainerBase Peter Mayes
John Pope , 26-Apr-07 Coimmon sense
Great article and fully agree with Don Taylor, and with Janet's comments.
Effectiveness of training and development is best assessed by the questions : What is this individual or group doing better now, or what more are they achieving which is of real value to the organization ? Was the gain worth the effort and cost? Could we have done this better or more effectively? Why did we wait so long before doing it ?
ROI is much better used when deciding how to allocate resources between different projects.
Meir Navon , 26-Apr-07 This discussion is off the point!
After working in the training arena for more than 2 decades and trying numerous times to calculate the ROI of training initiatives, I've come to the conclusion that we are looking under the wrong tree! I decided to learn from our "bigger brothers" in the organization – the Sales people. How do they know if a product is good or not? They simply:
1. Ask their customers if they are satisfied with the product. 2. Closely monitor repeated sales
And this almost all!
While being the CLO of a big bank, the most important indicator for me if a managerial course (the ones that are really hard to calculate ROI in) is a success was simply if the Customer (yes with a capital "C" – the one paying the bills) was happy with the outcome and would like to have another course scheduled for his employees.
These people know what they are doing! They have to present their P&L reports every month and if they want to invest the time and money to send people to our courses, this is the biggest proof of its efficiency.
All the rest is very nice for academic reasons (and BTW I teach in a university, so no mal intention here...) or for covering trying to the inferiority feelings many of us have towards the other divisions in the organization.
It'd be very interesting to me to hear your opinions on all this.
Dominic Johnson , 26-Apr-07 Be Careful of Premature Evaluation
Disappointed with Don’s article in the Training Zone bulletin. It suggests no successful experience in this area and serves to mislead others in a profession that needing to demonstrate tangible value. Jack Phillips’ principles can be easily applied even if the full methodology is not always practical. Unless L&D professionals understand their clients’ underlying business requirements and commit to developing relevant solutions, they are wasting everyone’s time/money. The discipline/skill to get to the business issues and develop an agreed evaluation strategy separates true professionals from those who are well-meaning but of questionable value. I’ve developed methods of isolation and impact reports that demonstrated - to clients’ satisfaction - relatively small interventions (e.g. performance-coaching 30 retail stores) produced value of >$1m per month. Adopting the Jack Phillips’ methodology added a week’s work yet significantly improved our relationship with the client. Their availability of good data - and the ease of isolation - helped. However, the general principles produce great results whenever I apply them. With the paradigm of proving business value, plus performance consulting skills and business acumen, Jack Phillips’ principles can be applied to any assignment. They won’t all result in an ROI analysis - Jack says this isn’t always possible/desirable – but the approach/intent always reframes the client’s view of your contribution and better informs one’s solution. Happy to discuss this with you at more length if you’re open to learning about this. In the interests of honest professional contribution ( your intent), please don’t decry as misguided that which you may not have been able to make work. I suggest, as the leader in Training Zone points out, this is an issue that isn’t going away. Business wants L&D to prove its worth and, until we do, our income from well-meaning “solutions” of intangible value are highly vulnerable.
Very interesting article despite the error, however. ROI isn't complicated at all, in my opinion. There is a simple formula that anyone with a high school education could calculate. I don't see why people make this difficult. It's not.
Tery Tennant , 25-Apr-07 ROI is just a tool . . .
And I have to 2nd this comment by Paul and add another. The focus on ROI and documenting the real value of T&D activities is a good one, in that for too long, many development programs weren't clearly tied to business outcomes. We've all seen way too much of the "flash-in-the-pan type of seminar training (often without very little analysis of the true issue needing to be addressed) which does little to change behaviors back on the job, let along have any lasting impact on the value of the organization. So we shouldn't dismiss what the essence of seeking the "Holy Grail" of ROI is really about - making sure our T&D activities are truly relevant. Otherwise, it often is just throwing money down a black hole.
But as is the case in many human endeavors, at first the pendulum swings too far to the other side. I believe this is what's happened with strictly applying the Phillip's methodology - but the quest for designing targeted programs that have real, demonstrated impact is certainly a good & noble cause. (And while we're discussing this, a far simpler approach is the baseline & value-added method of Paul Kearns). Tery Tennant
Paul Kearns , 25-Apr-07 ROI is just a tool ...
Donald and when you know its strengths and weaknesses, when to use it and when not to use it, it can make a huge difference to the way organizations learn. For a fuller, more up-to-date answer (Jack Phillips model was never convincing)I suggest you read 'Evaluating the ROI from learning' (not training) from the CIPD - and you won't need a calculator! Paul Kearns
Janet George , 25-Apr-07 Common sense prevails!
How great to see some clear thinking common sense on this topic! And a way of demonstrating value! Let's hope we see less of those holy grail discussions on ROI and more like this. Thanks Donald!
Tery Tennant , 25-Apr-07 The ROI of getting the ROI!
Great article, and a conclusion we've come to as well. We know of someone who worked with Phillips & drove herself near crazy with all the ROI formulations. The intensive measurement activity at her company failed, and she left.
While we continue to document the results of behavior changes & goal achievement through our programs, we are also very cognizant of not using a 5 pound hammer to swat a gnat! Plus, many of our clients are sold on the value of the training any way - if true behavior change actually takes place. We help participants set a baseline, and measure progress themselves. There is usually some good low-hanging-fruit monetary benefit as well, which can be shown to easily pay for the program (if anyone is that interested). Often, the demonstrated monetary benefit is just icing on the cake, as far as the client is concerned! Tery Tennant
Everyone wants an ROI study to prove the value of their training – or do they? Return on Investment is complicated and time consuming to prove and, argues, Donald H Taylor, usually only the training department is interested in the results.
