Measuring Success: Does the Learner Matter?

As I continue to complete a variety of different eLearning projects, I often look back and think about how successful these projects were. My tendency is to focus on issues like:

• Did the client want to strangle me during the project or at the end, did I get a group hug?
• Did we conduct a few more (e.g., 14) reviews than planned (e.g., 3)?
• Once completed, did my manager use the term “red” when describing the project’s margin?
• How many team members had to take PTO for PTSD as a result of the stress?
• Once completed, did the client’s e-mail to me include words like, “wonderful, effective, highly-interactive, etc.” to describe the course or words like, “boring, confusing, problematic”?

All of the comments above beg the question, “How do you measure success on an eLearning project?” And second, does the learner matter? Obviously, the latter is meant to be a rhetorical question to which you should respond, “Duh! Of course the learner matters.” However, my bulleted questions above point out a tendency that occurs often - that is, to measure success of the project and not of the learning that takes place as a result of the project.

I’d take a wild guess that most companies can show you a table that displays the margins they have achieved on various projects. But how many even track quantified measures of learning that occurred as a result of those projects and share that with everyone? My guess is not many.

In an article written back in 2003 by the eLearning Guild (http://www.elearningguild.com/pdf/2/110303mgt-h%5B1%5D.pdf), the author states that it’s not how often you train, it’s how well. To measure “how well,” he suggests using Kirkpatrick’s Four Levels of Learning Evaluation, which includes:

1. Level One - Reaction. Did they like it?
2. Level Two - Learning. Did they learn?
3. Level Three - Behavior. Did they use it?
4. Level Four - Results. Did it impact the bottom line?

As I think back at all of the eLearning projects I’ve managed, I’m fairly sure I would not give myself an “A” for relying on Kirkpatrick’s levels when measuring the overall success of the project. But in order to conclude the projects had been successful, I believe I should have.

Now, let me return to the implications of the title for this post. How should success be measured? And does the learner matter? Better yet, how do you currently measure success on your projects? And in those measurements, did the learner matter?