How would you feel about being rated by your delegates and clients online? Garry Platt argues that an independent feedback website could be more valuable and telling than many training qualifications.
High on the agenda at the moment is the necessity for trainers to maintain their professional development so that their skills keep pace with the market place. Various training bodies offer refresher training and various approaches to accreditation that 'guarantee' competence. A flourishing qualifications industry is growing up in this field all making claims for their validity and legitimacy and covering all aspects of development. From basic stand up skills programmes through to advanced Feng Shui'd programme design.
But what really counts, a qualification or something more pragmatic like a consistent record of positive feedback and results for clients? Do I really care if someone has a got a CTP from CIPD or a Certificate in Training Delivery Skills from TAP, or a Certificate in Training & Occupational Learning from ITOL or indeed an Advanced Diploma in Canine Grooming from the Bradford Academy of Dog Care?
"There might be two assessment areas: client or business feedback and student/participant feedback... This data would carry significantly more credibility than any qualification or accreditation no matter how well intentioned these are."
In the USA a strategy has begun to develop which helps university students for courses have access to the evaluation and observations of former students. Lecturers and courses are named and the students grade them. It has helped promote some careers and blight others. For instance, who wouldn't want Ariadna Bogatsky as reviewed here for a teacher. Where as I'd have to think twice about taking Ronald Beall's classes for a semester. (For years I thought a semester was a kind of cheese cracker, how wrong can you be?)
In commercial tenders we are consistently asked for references from previous clients. And clearly how your previous customers judge their experience of working with you carries significant importance as it should. This adds to the view that a consistent record of positive achievement and feedback from clients probably carries more weight than any qualification. So what if there was a system for publishing this information via the web in a similar way to the CourseReviews website? Such systems can be successful; the equivalent of this is the hotel star grading system by which a hotel is classed as 2, 3, 4 or 5 stars. But a visit to the Trip Advisor website where guests and clients undertake the reviewing will reveal that many 4/5 star establishments leave a lot to be desired, whilst some 2/3 star locales appear to offer excellence above and beyond their official grading. I know this information influences my choice of places to stay.
It would be possible to create a registration process and have an assessment marking process against set criteria with opportunity for comments. It would be a powerful marketing tool but it would also expose those trainers who stopped developing towards the end of the Triassic or have simply gone 'off the boil'.
Trainers who failed to direct students towards the review site could find their grading slipped as the number of reviews diminished and time increased since the last submission. There might be two assessment areas: client or business feedback and student/participant feedback. That way the overall impact and bottom line assessment of organisations might be viewed as well as the classroom-based skills. This data would carry significantly more credibility than any qualification or accreditation no matter how well intentioned these are.
Trainers could register with such a site and then direct clients and participants to their individual review section. Imagine the power of this as a marketing tool if you achieved consistently good reviews. This is feedback straight from the horse's mouth; much of the promotional material we typically read originates from the horses other orifice. Trainers could use such a reviewing system as a kind of quality mark which is uniquely not driven by intermittent testing or a reassessment process. An overall assessment would obviously have more credence based on the quantity of people submitting reviews. Potential clients could quickly check out the profile of trainers and also see how previous employers assess the impact and results they achieved.
"It would be a powerful marketing tool but it would also expose those trainers who stopped developing towards the end of the Triassic or have simply gone 'off the boil'."
I can see the consternation such a tool might create for some people. If the process took off the best would quickly be identified, where as others might find it difficult to sustain a career.
A qualification can be important to establish that the individual concerned had at one time the capability to retain knowledge and demonstrate to a competent level a particular range of skills. It can be a good indicator of dedication and application. But that's all a qualification means. I am reminded of a university professor who had just watched a group of his students collect their PhDs on graduation day. She approached one of her favourite students and asked; 'Do you know what BS stands for?' The student looked back quizzically and responded. 'Yes, of course, Bachelor of Science.'
'No,' replied the professor, 'it stands for Bull Shit'.
'What does MS stand for?' The student hesitantly answered: 'Master of Science'. The professor shook her head. 'No, it means More Shit. And what do you think PhD stands for?' 'I've no idea.' The student answered, his hubris somewhat fading.
'Piled Higher and Deeper,' The professor replied.
And the pile is surely starting to get higher and deeper around here.
Garry Platt is a senior consultant at Woodland Grange specialising in management development and trainer training. He can be contacted on 01926 336621 or e mail: garry.platt@wgrange.com
Garry Platt , 04 November 2008 @ 06:45 AM Good Idea
Tery Tennant wrote: "The boss and participant together should establish what is needed to be achieved from the training. They should set a baseline measurement beforehand around the important indicators. Then afterwards the boss needs to be supportive - schedule follow-up measurement and accountability towards making the needed changes."
I think that's a powerful way to go Tery. One of my own strategies when dealing with clients who focus just on end of course review sheets is to educate and introduce them to the more important and relevant aspects of training evaluation and in that way raise the standard of the game for everyone.
Occasionaly managers don’t know any better because they’ve never been introduced to bottom line evaluation strategies. I find introducing these approaches creates a closer working relationship between me and my client. I give them systems and processes which work in conjunction with the training delivered.
I believe this is absolutely essential in the day to day practise of training. It’s also probably why I don’t encounter quite so many managers who aren’t interested in ROI analysis because it is part of my training process to introduce them to these processes where necessary. They are then much more empowered to make business driven decisions on training rather than being driven by urge and fad, they also learn to the true value of reaction level evaluation.
Tery Tennant , 03 November 2008 @ 18:54 PM To rate us properly, the manager and participant should be involved in establishing the baseline
As L&D professionals, part of our job should be to train the client on good effectiveness measurements!
Here's the logic: L&D shouldn't happen in a vacuum, therefore the participant's boss should be fully in the loop. The boss and participant together should establish what is needed to be achieved from the training. They should set a baseline measurement beforehand around the important indicators. Then afterwards the boss needs to be supportive - schedule follow-up measurement and accountability towards making the needed changes.
So in this way the client (both manager & participant) would truly know how to rate our work, and it would move things away from being such a subjective endeavor!
Garry Platt , 03 November 2008 @ 11:29 AM Yes, No, Maybe.
Nik Kellingley wrote: "And I quote... 'There might be two assessment areas: client or business feedback and student/participant feedback.' Might is not would be, it's definitely might."
The ‘might’ Nik was actually in reference to the entire hypothetical proposal not to my view of what should be included as I think the following full quote confirms:
‘There might be two assessment areas: client or business feedback and student/participant feedback. That way the overall impact and bottom line assessment of organisations might be viewed as well as the classroom-based skills. This data would carry significantly more credibility than any qualification or accreditation no matter how well intentioned these are.’
Regardless of this I think I have made it absolutely clear that I propose that both elements should be present and one is more important than the other.
Nik Kellingley continued: “To improve the profession it would be much better for learning professionals to genuinely encourage learning based on success/failure of real life examples. Than to move forward with what is in effect an online happy sheet particularly as the forms would (by necessity) need to be a one-size fits all method for completion...”
The former comment I absolutely agree with, it doesn’t happen though. I believe my proposal might help us move just a little distance towards that goal. The latter comment is a little wide of the mark I think, though I do share Nik’s concerns about any ‘one size fits all’ system. In the absence however of *anything* else and in the presence of an increasingly fraught and fractured training qualification framework some compromise might be appropriate. Perhaps some variants to organisational assessment might be considered where the trainer has played a diminutive or more comprehensive role and a selection is made accordingly. I don’t think it is impossible to create a straightforward analysis which encourages managers to reflect on and assess the extent to which the trainers/developers have been successfully or otherwise. Speculatively, here’s 5 principal areas:
1. Recognised organisational priority result areas. 2. Analysed where shortfalls existed due to an absence of knowledge or skill. 3. Delivered a suitable and effective response. 4. Contributed to measuring the impact on the organisation. 5. Achieved a positive outcome for the organisation.
The proof of the pudding will of course be in the eating (if it ever gets baked). I would certainly suggest that it would be an improvement in the current situation and as a side issue would of course lead to identifying what courses and training qualifications correlate strongly with successful organisational training, wouldn’t that be interesting to find out?
Nik Kellingley , 03 November 2008 @ 10:55 AM Yeah, but no, but...
And I quote..."There might be two assessment areas: client or business feedback and student/participant feedback."
Might is not would be, it's definitely might. And more to the point while I think enabling the use of online feedback for particpants is easy to do - you can even set up in your learning environment if you're smart.
Compelling clients to deliver feedback is not as easy and in many cases they may not wish to make that feedback public.
I agree with Garry (it has to happen occasionally) that there are moves towards outcomes based training from some companies. Equally there are many companies that don't - this is why fire-walking and the like still thrive.
Juliet is also right that many companies do not measure training effectiveness by any other means than participant feedback. So it would likely reflect the electronic happy sheet.
(I too did a project for Network Rail a while back but they are an exception in my experience and not a rule when it comes to outcomes.)
The truth of the matter is - in the last 3 months on training zone there has not been one single case study from a trainer, where there TNA deviated (or even matched) from their expectations of result. No learning points to discuss in how the methodology met/exceeded/failed to meet the requirements of their TNA.
To improve the profession it would be much better for learning professionals to genuinely encourage learning based on success/failure of real life examples. Than to move forward with what is in effect an online happy sheet particularly as the forms would (by necessity) need to be a one-size fits all method for completion... Nik Kellingley
Garry Platt , 03 November 2008 @ 09:21 AM Shared reality?
Juliet LeFevre wrote: Garry, we do hear you, we just see it working differently to the theory. I doubt that they'd listen to this, they only pay attention to user feedback, also your average delegate and their manager (often the one booking the course) also only pays attention to immediate feedback.
“They only pay attention to user feedback.” "their manager (often the one booking the course) also only pays attention to immediate feedback."
In my experience Juliet this is not the case, whilst not denying it might be yours. It is true that in many instances reaction level feedback is the only thing trainers provide so what else can they look at? However when something of more weight and significance is offered to them they grab it with both hands, that’s my experience. The possibility of highlighting the impacts of trainers on the bottom line of businesses would be of extreme interest to virtually every manager commissioning training I have encountered in the last 10 years. I don’t think managers are so blinkered as to ignore this source of information. A national scheme that provided client assessments on the extent the trainers have been able to identify and provide suitable responses which have made real and significant impacts on their respective organisation's success would be extremely useful. In much the same way the JD Powers survey identifies true value in various types of car.
Juliet LeFevre finished with: “They'll think its like ebay - arguably they regard happy sheets like this already. Its only trainers and training managers that try and prove long term value and Level 2/3/4 evaluations. The man on the street isn't interested.
I don’t understand the ebay reference so can’t comment. ‘The man on the street isn’t interested’. Again that might be your experience, but it is not true for me. I’m currently working with Alps Electronics, the MOD and Network Rail on significant contracts all of the Departmental Managers want to see something more than end of course review sheets and all of them have stipulated workplace outcomes that are to be achieved as results of the training contribution. This is now typical and I suspect and hope a growing trend spurred on by the economic climate we are all facing over the next 2 years.
As I said before this system would only succeed if it provides information of real benefit to clients. Juliet’s assessment that line managers would ignore or be disinterested in the assessment of work place benefits that trainers achieve might be correct, frankly I doubt it.
Juliet LeFevre , 03 November 2008 @ 05:19 AM Reality
>>>To be absolutely clear; managers from within the organisation would make assessments of the training, designed, developed or delivered by the individual under assessment and this would be the essential factor of assessment. Not how much they were liked by their learners but rather what impact they made to the business areas the training was designed to impact on.>>>
Garry, we do hear you, we just see it working differently to the theory. I doubt that they'd listen to this, they only pay attention to user feedback, also your average delegate and their manager (often the one booking the course) also only pays attention to immediate feedback. They'll think its like ebay - arguably they regard happy sheets like this already. Its only trainers and training managers that try and prove long term value and Level 2/3/4 evaluations. The man on the street isn't interested.
Garry Platt , 02 November 2008 @ 20:18 PM One More Time
I would agree with Nik Kellingley’s observations and Juliet LeFevre’s comments if the system worked in the way they have re-defined it, but it doesn't. My proposed model addresses the issues they raise. Let me emphasise this by repeating my previous comment:
“I agree, too much emphasis on the personal impact of the trainer would result in a deleterious emphasis on trainers with plenty of style but not necessarily much content. That is why I think its essential (and I emphasised) that this process has two sides, a reaction level (which is relatively important) but secondly and of absolute importance a business impact section where organisations/managers assess the merits and impacts of the development delivered. It would be fatal as you quite rightly point out to have one without the other.”
To be absolutely clear; managers from within the organisation would make assessments of the training, designed, developed or delivered by the individual under assessment and this would be the essential factor of assessment. Not how much they were liked by their learners but rather what impact they made to the business areas the training was designed to impact on.
Nik Kellingley , 01 November 2008 @ 18:37 PM Measuring the wrong things...
There was a point in time when I thought that this kind of thing would be an excellent idea and would have wholeheartedly supported it.
But... I'm not sure learners are qualified to judge the usefulness of training in particular I'm not sure they know what to measure to gauge things.
Real World Example Number One - I was speaking to someone the other day who felt the best training course that they'd ever been on was an outward bound team building course. I said I found this a little surprising as I felt these courses were usually pretty poor in delivering an outcome that was quantifiable and asked what they'd done with the learning in their team? After a minutes reflection they admitted that they'd left the job a few days later and in fact while they'd enjoyed the course enormously it had, had no significant impact on their behaviour since...
Great Rating for Trainer - even though it's a Lousy Course. Is what you'd get here.
Real Life Example Number Two - One of my own, I had a trainee who wasn't progressing and I gave them some very hard feedback in that respect. It was clear the trainee hated me at that point and that they carried this through to the end of the course. Four years later, I was out having a drink in Manchester when who should approach me but this particular trainee - who told me she'd hated me for months after that, but was now very grateful for the intervention and that she felt her current success was down to the course and that intervention.
Result: Lousy Feedback on Website, Course that achieved it's objective
And so on. All this does is enable a measurement of how much people "liked" a course, and not how much difference it actually made. That's why hotel ratings are useful - because liking a hotel is what it's all about. But liking training is not the be all and end all, achieving your objectives are. Sometimes to be a great trainer you have to set aside your desire to be liked in order to help someone else achieve. Now most of my delegates have never seen the hard side of me - nor do I want them to. But occasionally it's essential.
We as trainers need to go much further than public online happy sheets. We need to share our successes and more importantly our failures for peer review. We need to treat our profession like a science and not a church of belief... Nik Kellingley
Garry Platt , 30 October 2008 @ 09:51 AM Two Sides
Juliet LeFevre wrote; “If this is successful, with time one 'format' will win through - ratings will win and qualifications will be rendered redundant. Fun, zingy trainers will be de riguer.” [&] “Data Protections issues aside doesn't this give too much weight to Level 1 evaluations?”
I agree, too much emphasis on the personal impact of the trainer would result in a deleterious emphasis on trainers with plenty of style but not necessarily much content. That is why I think its essential (and I emphasised) that this process has two sides, a reaction level (which is relatively important) but secondly and of absolute importance a business impact section where organisations/managers assess the merits and impacts of the development delivered. It would be fatal as you quite rightly point out to have one without the other.
As for Data Protection I think that is capable of being surmounted, other organisations do.
I also share some of your concerns about this being an approach belonging to one group or association; I think it needs to be open to everybody as the two other sites I have referenced are for it to have any National viability.
I think inevitably the market would drive the success or otherwise of this process, dependant on three key issues: 1) Does it provide valuable, reliable and evaluative data on a trainer’s impact on the organisations with whom they work? 2) Do the criteria of assessment reflect what organisations need and want to know about? 3) Does it work? Does it provide a better insight into the effectiveness of a trainer than any award, qualification, allegiance to a charter or accreditation?
Juliet LeFevre , 30 October 2008 @ 09:31 AM Yeah but....
You wouldn't catch me paying to join an organisation and then forking out for the libel claim because of its own procedures, especially if I have to follow its own qualifications route also - talk about one cancelling the other.
If this is successful, with time one 'format' will win through - ratings will win and qualifications will be rendered redundant. Fun, zingy trainers will be de riguer.
With even more time we'll have a enquiry into the ratings agencies or those bodies running them. (Echoes of Financial Crisis).
Those that are sponsored will have conflicts of interest, and fight for prominence pushed by the sponsor.
Data Protections issues aside doesn't this give too much weight to Level 1 evaluations?
Peter Mayes , 30 October 2008 @ 09:07 AM The plot thickens
Garry
Thanks for the release for the use of the idea; my mind is off on all sorts of tangents trying to think how this idea can be implemented for the masses and used to support our own accreditation scheme.
I have had an initial estimate for development of a mulit question rating system that plugs into (becomes part of) Trainerbase. For it to work all parties would have to be registered to post and receive a ratings (would help with the policing by eliminating fictitious ratings). The system would of course need funding for maintenance and policing and unless this was part of TrainerBase's membership we would have to source the money from elsewhere; if anyone has any sponsorship ideas on this I'd (or Barry our Commercial Manager) would be interested (01239 711544).
Of course a sponsorship deal could enable us to deploy this within a separate site (whilst still being 'powered by' and part of TrainerBase).
Hmm and my mind is into domain name registration mode now; the plot thickens.
Garry Platt , 30 October 2008 @ 06:48 AM Developments
Peter Mayes wrote: “Consider; an individual goes on a Train the Trainer course and their expectiations(sic) are not met because they failed to research the type of establishment they were attending and the scope of the course. How would the independent web site police a posting by the attendee saying that the course was useless?”
Well I think the trainer could have the right of reply, in fact I think that would be almost essential. In such a case it would be possible to highlight the shortcomings or absence of the individual’s responsibility to undertake some preliminary research before attending a course. Using this approach vexatious, unwarranted or unfounded criticism could be exposed.
It would also be true that the libel laws would still stand and anyone found making unsupportable claims might well find themselves on the end of a legal writ.
Peter Mayes added; “And as I recall (way back in the dark ages), a freelancer is only as good as their last job; what happens if that job is reported as a duff one but the results are outside the providers own control.”
The overall assessment I would suggest could be against a 12 month rolling period and one bad review against a significant number of positive assessments would not necessarily make a huge impact, if any, on their overall assessment. One or very few negative reviews would stand out like a sore thumb as untypical. I believe that most people recognise that you can’t please everybody all the time, but you should be able to satisfy most people most of the time.
Peter Mayes concluded with; “Within TrainerBase's CLP accreditation we have a multidirectional review procedure and the requirement for a candidate to provide evidence of what they do. Our expectation is that this offers a level of confidence amongst purchasers that the trainer, course developer (such as Sheridan) or training adviser; does what they say on their profile, brochure,”
That’s better than most systems Peter. A constant flow of assessment rather than a one time or intermittent review would give me even more confidence in the working practises of the individual/s concerned.
You may of course do what ever you like with this concept Peter. I believe for this kind of approach to flourish however it would need to be available to anyone and not just to members of a limited group. In the same way the teacher review sites are open to everybody.
Peter Mayes , 29 October 2008 @ 15:35 PM The truth hurts
Garry
I think this is an excellent idea (in principle) and will (for our members own bnefit) look into how this could be added to the Trainers' profiles on www.trainerbase.co.uk (we have testimonials already). TrainerBase ltd is a not for profit trade association and as such have some level of independence.
The caveate of course is that the truth can sometime hurt. There is a court case eminating out of Ebay regarding a less than favourable testimonial posted by a purchaser. And as I recall (way back in the dark ages), a freelancer is only as good as their last job; what happens if that job is reported as a duff one but the results are outside the providers own control.
Consider; an individual goes on a Train the Trainer course and their expectiations are not met because they failed to research the type of establishment they were attending and the scope of the course. How would the independent web site police a posting by the attendee saying that the course was useless?
Within TrainerBase's CLP accreditation we have a multidirectional review procedure and the requirement for a candidate to provide evidence of what they do. Our expectation is that this offers a level of confidence amongst purchasers that the trainer, course developer (such as Sheridan) or training adviser; does what they say on their profile, brochure, or other marketing literature (regardless of whether they have any qualifications); TB would (and does) recommend them.
Right; off to write a technical brief for our development team:)
Garry Platt , 29 October 2008 @ 13:47 PM Good Point
Sheridan Webb wrote: “I feel that there would need to be an option for businesses to rate the effectiveness of non-traditional training solutions such as e-learning, self-directed training and distance learning that is designed for them.”
I agree, this area could be covered by the ‘client or business feedback’ section which would outline (regardless of the medium) the experience of the impacts of the development strategy undertaken and their satisfaction or lack of it from the results.
I appreciate that if you don’t deliver too much then the ‘slippage’ factor might impact on your overall training/delivery score. This is all hypothetical; perhaps in your Biography section this would be explained and you might not register for the section covering participant experience feedback analysis.
I did consider this primarily as tool for reviewing the performance of delivering trainers and consultants and more importantly their respective impact on organisations. An individual who does this constantly and consistently should (in my opinion) receive a higher rating than someone who undertakes work less frequently. A batting average should be linked to frequency of play otherwise the results could be somewhat distorted.
This whole concept is a notion of mine and you raise a fair point and I believe the issue could be overcome to retain fairness.
P.S. There are two, 2, II, Deux R's in Garry - not that I am at all sensitive about it. ;)
Sheridan Webb , 29 October 2008 @ 13:26 PM Great idea...but...
Gary,
I think that in principle, this is a great idea. However for some trainers (such as myself) for whom delivery forms the smallest part of my work, allowances would have to be made. If I only deliver a dozen courses a year, no matter how good they were, I could not get a good rating if your rating 'slips' with infrequent responses.
I feel that there would need to be an option for businesses to rate the effectiveness of non-traditional training solutions such as e-learning, self-directed training and distance learning that is designed for them. Choosing the right designer is an important as choosing the right deliverer, and they are not necessarily the same person!
How would you feel about being rated by your delegates and clients online? Garry Platt argues that an independent feedback website could be more valuable and telling than many training qualifications.
"There might be two assessment areas: client or business feedback and student/participant feedback... This data would carry significantly more credibility than any qualification or accreditation no matter how well intentioned these are."
