This is the complete transcript from the first EvaluationZone workshop run by Paul Kearns on January 10 2002.

return to the EvaluationZone workshop area.

Stephanie Phillips Hi Meghan, Sue, Donna.
Stephanie Phillips Thanks for coming along.
Stephanie Phillips We'll just wait a few minutes to see who else turns up.
Stephanie Phillips Hi Ian, Perry.
Stephanie Phillips Thanks for coming.
Perry Williams Hello!
Stephanie Phillips Hi Angela.
Stephanie Phillips Looks like it's going to be a busy session!
Stephanie Phillips We'll wait a few more minutes, then get started.
Angela Holroyd Hi Stephanie.
Paul Kearns Hi everybody.
Stephanie Phillips Hi Dave - thanks for coming.
Stephanie Phillips Okay, looks like most people have arrived. If there are any latecomers, I'll introduce them later.
Stephanie Phillips Thanks and welcome to everyone to this first EvaluationZone workshop, hosted by Paul Kearns.
dave qh Hi Stephanie, took me a bit to discover how to respond.
Stephanie Phillips Paul is online for the next 3/4s of an hour to talk about some of those ROI issues which keep cropping up...
Stephanie Phillips...and to explain a bit about how EvaluationZone is going to work.
Stephanie Phillips It'd be great if everyone could just do a quick introduction - let us know who you are, what you do and what your particular interests are, so Paul can address them.
Donna Hosie I work in the pharmaceutical industry. Interested in evaluation generally.
Stephanie Phillips Thanks Donna.
Sue Press I work in IT, interested to know how to get the business excited about training.
Stephanie Phillips Great - thanks Sue.
Angela Holroyd Angela Holroyd Freelance Training Consultant. My particular interests are how you actually calculate ROI, and how you can use that to persuade companies to invest the money in training in the first place.
Fran Burgess Hi, I am a freelance trainer and like others any tips on evaluation most welcome.
Ken Teal Portsmouth based, performance improvement consultant - main tool is measurement (SPC in manufacturing environments) most training is delivered as part of a project implementation.
dave qh We are a learning outsource company for proprietary technology training.
Perry Williams I develop materials, for public sector as well as corporates; increasingly clients are concerned about ROI, and want to advise them.
Mike Wiltshire I'm a specialist provider of financial training, under the name MoneyTalks Training
dave qh My interests are design to delivery ratios.
Ian Lever Hi - I'm Technical Director of LEVER Technology Group plc. We provide technical training to the Telecommunications industry and to major corporates. We increasingly need to calculate and demonstrate ROI to our customers; I'm also interested in evaluation.
Stephanie Phillips Great stuff - thanks. Hi Antoinette, George, Eszter - just doing introductions, do join in.
Meghan Bailey I am working as a training/learning specialist - interested in issue generally - especially measurement methods you can share
Eszter Molnar Mills Hi, I am training manager at a student's union - increasingly interested in evaluation as a whole.
Antoinette Gaskell Hi. I'm the European Training Director for an IT consultancy. We're a relatively new company and I want to establish some key metrics/evaluation systems to monitor effectiveness.
George Parker I'm the training and development mgr for Dixons Customer Contact Centre - also have with me Jill & Darren Mgt Dev Trainers - interested evaluation of soft skill training.
Stephanie Phillips Thanks all - some useful background information there. Let's bring Paul in - over to you!
Paul Kearns Thanks Stephanie - I think we should look at the basic calculation first them work from there. The first thing in ROI is to establish the benefit and try to assign a £ sign, the cost side of the equation is simpler but not of such importance. Has anyone got a piece of training they would like to discuss the £ sign benefit for? If there are no takers on this let us look at Ian's technical training this causes an immediate problem for ROI. Does anyone know why?
Donna Hosie ? Technical competence varies.
Ian Lever I have to cost-justify the training, but I have limited ability to measure the employee's increase in performance.
Antoinette Gaskell No. But I'd be keen to know because that's our focus too.
Paul Kearns The problem with most technical training is it helps people do their jobs but there is no sense of business improvement. Let's take differences in competence.
Corinne Wilhelm Qualitative or quantitative?
Paul Kearns If someone is more competent then to get the 'benefit' we must talk about their efficiency for example. We will always talk about quantitative Corinne. Take 10 employees and ask the simplest question how much do they cost us. If we train them and they become 10% more efficient we can save one employees costs - the benefit becomes the cost saving. What other benefit will we get from competence to put into the ROI equation?
Ian Lever Staff motivation, retention.
Antoinette Gaskell But if you were to train an employee in a new technical skill area - a programming language for example -how would you measure change in efficiency?
George Parker Paul where does the calculation of the trainer resource come into this theory?
Perry Williams How could the company tell if the employees are 10% more efficient (unless in something like sales, where cash directly attributable to individuals).
Corinne Wilhelm But how are you measuring their efficiency - with for example being more effective in an intercultural negotiation? Different negotiation, different partners etc...?
Antoinette Gaskell But there are so many other factors affecting motivation and retention how would you measure the training element?
Paul Kearns Let us apply it to a team of administrators. Let us say they do not know how to use a new IT system so we have to train them. This is an either or situation - either they can use the system or they can't.
Corinne Wilhelm ok
Paul Kearns If they can after training all well and good. If they can't there is nothing to measure.
Corinne Wilhelm Yeah but life isn't so black n white is it - different technical affinities, experience etc.
Paul Kearns This goes back to my first point that basic/technical training is not the best type of training to ROI. Let us pick up on some of the issues already raised.
Angela Holroyd I agree with Corinne, its too black and white, there is so much in between.
Corinne Wilhelm So what types of training are better suited?
Paul Kearns The most suitable type of training to ROI is where a business performance gap has already been identified.
Ken Teal Not sure tech trg is the best example - it comes under MUST HAVE and the return is on the policy decision or the cost of failure.
Corinne Wilhelm For example?
Paul Kearns In the case of the administrators the new IT system is introduced to reduce operating costs by, say 100K. Once you start from this perspective you can put a potential benefit into the ROI equation. If the training is not focused on a business gap it cannot be subject to a straight forward ROI.
Corinne Wilhelm So we have to get to the business gap up front?
Paul Kearns You got it in 1 Corinne.
Corinne Wilhelm Are people always prepared to give you that?
Paul Kearns We can deal with more difficult cases later. This is where we need to excite the business. If we cannot get ROI benefits agreed up front what chance have we of measuring them afterwards.
Corinne Wilhelm Fair comment.
Paul Kearns Ken is right basic training causes many problems for ROI - because it should not be ROI'd.
George Parker So if we are training a number of managers to improve their coaching skills and/or communication skills - any ideas on how we evaluate if their newly acquired skills have a ROI?
Paul Kearns This is where many trainers get their knickers in a twist!
Paul Kearns Has anyone got a sales programme or customer service programme?
Perry Williams Paul, do I understand you right, you not only need to agree ROI benefits up front, but the business already has to have identified a gap?
Paul Kearns Yes Perry otherwise why do the training?
Paul Kearns Are we all getting confused already?!
Corinne Wilhelm But the company might not realise what training is really needed, we need to advise them.
Paul Kearns Right again Corinne, but where do you start your TNA?
Fran Burgess I am currently putting together a programme to train 36 sales staff who will be selling 3 new financial products.
Paul Kearns Lets go with Fran's idea.
Corinne Wilhelm Excellent - an actual example is what I need I think.
Paul Kearns Before you do the sales training the main ROI question is what financial benefit will result. This should now be easy to calculate - it will be the projected sales on the new products.
Anne Reed Hi, I'm Anne Reed, responsible for getting NHS staff in north + mid Hampshire using computers. I'm trying to get staff doing the European Computer Driving Licence. I need to show in advance what the benefits might be and then show we have achieved them afterwards. Any suggestions?
Paul Kearns Do you have any ballpark figures Fran? If you have trouble finding the figures then why not look in the Sales Directors business plan? What if the SD doesn't have any figures what is their commitment to the training? Are you looking for 1% or 10%?
Fran Burgess I do not have the actual projected figures for my client but would estimate approx. £2.5 million.
Paul Kearns Great - this looks like important training!
Corinne Wilhelm Fran, help him out a little..
Paul Kearns Before we move on, ROI also starts to prioritise any training worth more or less than 2.5M.
Paul Kearns Next question is the cost side of the equation. How much will the training cost? We need this to work out the ROI?
Fran Burgess This is a three year programme and will cost the company approx. approx. £20k per year.
Paul Kearns The immediate ROI on that is enormous if we attribute all sales to the training, but we cannot do that because everyone will argue that advertising etc. has a part to play. So ROI tries to calculate the difference the training makes. Let us assume it makes 1% difference - 1% of 2.5M is 25,000.
lesley rooney On what basis is the assumption of 1% made?
Antoinette Gaskell But Paul how do you determine whether it is 1% or 10%?
Paul Kearns Over 3 years this would give a 25% return. Is that good enough? How do we answer this? Any offers?
Ken Teal Paul, isn't Fran's case tech trg for sales people? The "return" has been pre-calculated by the Board as a policy decision to go with new products. Are we to let some salespeople go out without training so that they can act as a "Control"?
Paul Kearns That's one possibility, Ken.
Paul Kearns Those of you who are finding this difficult try the 3 Box System after the workshop.
Corinne Wilhelm Sounds a bit scary to me?
Fran Burgess In this case, I will be able to measure the trained staff against the non-trained - if this helps!!
Paul Kearns It's not scary at all Corinne - quite the opposite! Here's a real example. A large retailer asked me to ROI a programme costing £1.4M. I asked them what the expected benefit was - The answer was no one had worked it out. I said, "why are you doing the training then?" The answer was because it would 'improve customer service'. I said, "what does that mean?". They said they would have happier customers. I said, "what do happier customers do? Do we want happy customers or customers who buy more from us because they are happier?"
Corinne Wilhelm Give me your money honey!
Fran Burgess We want them to buy more and tell their friends.
Paul Kearns This leads us to measure customer service in terms of improved sales.
Paul Kearns Exactly, Fran. But the benefit can only ever be measured in sales/cost or similar £ sign.
Perry Williams Suppose the organisation is public sector, and the "customers" are non-paying members of the public. How can one approach quantifying the benefit then?
Paul Kearns With basic training though Ken is right.
karen mckenzie Can you measure ROI in a preventative way, i.e. not having problems?
Anne Reed What if you are not in a commercial business. How in health care can I prove the benefits of IT training.
Perry Williams Like Anne says.
Paul Kearns Let us assume the co wants minimum standards of customer care then. This is a different proposition altogether. This is not about improving £ signs immediately so is not so suitable for ROI. What we need for basic training is validation. So if we agree to answer the phone in 4 rings we need to measure that.
Anna Walther You say the benefit can only be measured in sales / cost - in Customer service - why can't it be measured in Customer Satisfaction?
Ken Teal Karen has the best point - measure failure and its subsequent reduction.
Paul Kearns Because customer satisfaction on its own doesn't mean anything to the business. We can measure failure - how many customers did we lose? - but even then we need to look at the lost sales.
Corinne Wilhelm Perhaps you could simulate the failure somehow -?
Paul Kearns Yes Corinne but I don't recommend it!Imagine simulating failure in pilot training!
Perry Williams We had one example of performance management training where we argued ROI in terms of numbers of Industrial Tribunals they wouldn't be taken to.
Paul Kearns Control groups are a way around it, but why not pick an easy one to do your first ROI on? That way it isn't scary.
Corinne Wilhelm You mean like the brewers do in the UK? Mystery visits.?
Paul Kearns Yes.
Angela Holroyd One company I worked for measured customer satisfaction by the number of complaint calls it had per day.
Anna Walther But that's testing satisfaction.
Paul Kearns If you try ROI out on the right training I will guarantee you get a great result that will excite the business. Satisfaction has to be converted into £'s.
Corinne Wilhelm Some corporates tests call centres by the numbers of people hanging up. Someone hanging up goes to Bloomberg instead - ?
Paul Kearns That's probably a good validation measure, but not a good ROI one.
Anne Reed Can you use training need analysis to find out current level of skill, do the training and 6mths / 1 yr later do it again to see if they have improved. TNA to be task based i.e. can you format a disc - yes, just about, no , not sure.
Corinne Wilhelm Sounds good, Anne.
Paul Kearns Yes Anne, but it's not the easiest way to do ROI. Unless you get the £ sign before it's difficult afterwards.
Anne Reed I've done the TNA but don't know how to convert the improvement into cost saved in an NHS setting.
Corinne Wilhelm cost of bad press?
George Parker Paul, are you saying for soft skills it is about validating what you do rather than quantify ROI?
Paul Kearns The only measures you can use in ROI are financial. Anne needs to convince her people the training will add value in £'s. She needs their help and commitment.
lesley rooney in the public sector, the link between activity and £'s is not drawn tightly enough as yet to make this ROI a useable option I think.
Corinne Wilhelm Is this why e-learning is so attractive for sales?
Paul Kearns I don't know that it is. My first ROI on an e-learning project said it was not a good investment. This was not what the client wanted to hear.
Angela Holroyd I have been there too with e-learning.
Perry Williams In Anne's case, what are the alternatives if they can't format a disk? get someone else to do it? That can be quantified in terms of time/cost, which would be saved.
dave qh why was e-learning a bad ROI?
Paul Kearns Perry, I think Anne's case is actually basic training again. Not a good place to start in ROI. This short workshop has already raised a big issue.
Sue Press With anne's case the cost associated with asking a IT professional to do that simple task would be worth looking at.
Anne Reed E-learning is useful in that it saves the time and cost of travel. Many can be learnt at a computer either at work or home. I have found that professional staff want to do e-learning but admin etc prefer tutor led.
Anne Reed Thank you sue, I can work on that one!
Paul Kearns EvaluationZone will try and provide fuller answers. Can I suggest that we see this as a short intro only. EvaluationZone is not a quick fix.
Margaret Hartigan How do you differentiate between basic training and ROIable training?
Paul Kearns Already we have unearthed the issue of basic vs improvement training. This is not an easy one to unravel.
Anne Reed It will interesting to measure how many you get to your chat room when you charge for it!?
Paul Kearns That is why we suggest you try some of the exercises. When you try them out you will need to explore answers more fully. This is what the Forum is for. I think an understanding of evaluation takes a long time.
Corinne Wilhelm Remind us where to find the formula again please.
Stephanie Phillips There are lots of issues here which Paul can provide a wealth of information on through EvaluationZone.
Paul Kearns The formula is on the ROI page.
Paul Kearns There are a couple of questions as well
Margaret Hartigan Hi, I'm Margaret Hartigan - Sorry - what exercises?
Paul Kearns Have a trawl thru all the EZ pages.
Stephanie Phillips Hopefully everyone has had a chance to look at EvaluationZone - if not, visit it afterwards at https://www.trainingzone.co.uk/zones/evaluationzone
Paul Kearns My final word - I'm happy to take direct questions by e-mail.
karen mckenzie Thanks Paul.
Antoinette Gaskell OK. Thanks very much for your time Paul.
Stephanie Phillips The area is currently available for all members to try out until 20 January.
Stephanie Phillips Paul will be providing workshops like this regularly over the coming year to EvaluationZone subscribers.
Corinne Wilhelm Is it free until 20th then?
Stephanie Phillips Corinne - you'll be able to access the initial areas of the site free until then - but there's much more information to be added yet!
Fran Burgess Thanks Paul. You have confirmed that there are no easy answers to the ROI question!!!!!
Paul Kearns It gets easier as you learn, Fran.
Stephanie Phillips EvaluationZone will be a great opportunity to explore these issues in more depth over the coming year.
Corinne Wilhelm Thanks Paul and the rest of the contributors!
Paul Kearns Thanks to everyone for taking part.
Stephanie Phillips If you'd like more details on how to subscribe, send us an e-mail to [email protected]
Reg Yes, thanks Paul.
Angela Holroyd Thanks Paul and everyone else.
Paul Kearns My typing speed has improved immensely.
Stephanie Phillips Hope everyone's had an interesting session this lunchtime.
Corinne Wilhelm Excellent practice huh?
Margaret Hartigan Yes - thank you all.
Ian Lever Thank you Paul.
Stephanie Phillips Okay, let's wind up today's session. Thanks again to Paul for such an informative workshop.
Stephanie Phillips We hope to see many of you at the next EvaluationZone workshop next month.
Ken Teal Thanks Paul - I missed the last bit - the Gas man arrived to fix the heating!
Stephanie Phillips Okay everyone, let's finish the session now. To leave the workshop, simply close the window down.
dave qh Thanks folks.