-
Home
›
- Articles ›
- Approaches to measuring effectiveness and ROI
Approaches to measuring effectiveness and ROI
Profile's principles of "Plan, Do and Review" depend on organisations evaluating activity and establishing the value that it has contributed as part of their planning cycles.
How can you measure the value of something as intangible as good people management or effective learning? It can be difficult to achieve precise statistics, and a purely numerical approach may not provide the information you really need about how to improve performance. Profile suggests you adopt a variety of measures and techniques, looking not just at immediate feedback and perceptions but at the longer term impact.
In his classic “Techniques for Evaluating Training Programmes”, academic Donald Kirkpatrick set out a simple four level framework, to distinguish the different ways in which the overall impact of training – not just the short term perceptions - might be assessed:
|
Level
|
Description
|
|
1. Reaction
|
Assesses trainees' opinions of the programme, including the content, speaker, schedule etc., typically via evaluation forms or comment sheets. This level is important as companies often modify their training curricula based upon participants' feedback.
|
|
2. Learning
|
Measures the learning of principles, facts and skills and the adoption of attitudes that were specified as the training objectives to be achieved by the programme.
|
|
3. Behaviour
|
Measures the extent to which there are long-term changes in employees' behaviour and the aspects of job performance that are related to training objectives — "transfer of training".
|
|
4. Results
|
Gauges the impact of the training programme on business variables such as higher productivity or reduced costs. Relates training outcomes to organisational objectives and effectiveness criteria.
|
US studies have shown that while organisations generally do evaluate training at level 1 – feedback forms and the like – only a handful conduct follow up evaluation over time to identify the role the training has had in improving performance.
Increasingly, large organisations are challenging themselves to demonstrate the return on investment (ROI) of their learning and development and people management activities, to the extent of calculating the cash or percentage gain. This can be useful for comparing the return from different activities. For example: a training course vs a programme of secondments, since ROI converts them both to a common unit of business performance.
As an overview, estimating ROI or cost/benefit ratios can involve the following steps:
1. Collect data on results – actual changes in business performance
Gather data on the performance of the organisation over time, whether this is financial, operational or the perceptions of staff and customers
2. Isolate the effect of your development/people management activities
As far as possible, estimate how much of the change in business performance can be attributed to activity in question, as opposed to other influences. Is there a control group (i.e. a group of people/customers not exposed to the activity) which you can compare to those who were exposed? Was there a consistent trend before the activity was introduced? Does staff/customer feedback about the activity suggest it has had a major effect?
3. Convert the effects of the activity into a value of improvement
Once you know how performance has changed, and how much of the change is attributable to the activity, convert this into a business value such as time saved, quality gained, or more usually, money. For example, if staff turnover is reduced by 10 people per year, and filling each vacancy previously cost £2000, and a retention scheme accounts for 50% of the improvement, then the total value of the improvement is £10,000.
4. Work out the total value of the investment in the activity
Including both the cash amount spent, and the investment of time. In the previous example, the retention scheme might have cost £3,000 to set up plus the equivalent of £2,000 in staff time to run.
5. Compare the value of your investment in the activity with the value of the improvement attributable to it
Divide the ‘benefit’ by the ‘cost’ – in the case of the example, £10,000 divided by £5,000 gives a return of 200% or £2 for every £1 invested.
While the idea of demonstrating cash value is appealing, identifying relevant data and disentangling the impact of different factors can be extremely difficult, and will often come down to best guess estimates. Even sensibly estimated returns can be useful though, demonstrating that the organisation has not only evaluated its performance, but tried to assess value for money offered by different approaches to improvement.