I recently had the good fortune of attending a learning measurement workshop through LTEN, led by Steven Just of Princeton Metrics. Sitting in that virtual room populated by serious, experienced life sciences learning and development professionals, something landed harder than I expected. Not the content itself, which was excellent. It was the question underneath all of it: not how to measure, but what’s worth measuring—and why?
Formal workplace training as a discipline traces back to World War II, when the pressure of wartime manufacturing forced organizations to systematize how they prepared people for complex work. The Association for Talent Development—the field’s primary professional body—was founded in 1942. Donald Kirkpatrick introduced his four-level evaluation model in 1959. We’ve had nearly a century to figure this out.
And yet, here we are, still asking the same foundational question.
What the Data Can’t Do on Its Own
Most serious learning organizations are measuring. They’re running assessments, collecting survey data, and building dashboards. The real challenge is connecting what L&D can measure to what business leaders need to understand. This is harder than it sounds.
Tyler Vigen’s now-famous Spurious Correlations project demonstrated that the per capita consumption of margarine strongly correlates with the divorce rate in Maine. Or that the arson rate in Alabama correlates with the number of library assistants in the same state. Even though the data points line up (and are very entertaining), the logic doesn’t. This happens in L&D, too: completion rates trend up in the same quarter that sales performance improves, and—voila!—the dashboard looks an awful lot like proof of success.
According to Deloitte research, 95% of L&D organizations don’t excel at using data to align learning with business objectives, and 69% lack the skills to link learning outcomes to business results. That’s not a failure to measure. The data is there. It is, however, a gap in translation, and it points to something more structural than a dashboard can fix.
You Can Measure Almost Anything
Before xAPI, most learning data was captured by SCORM—a decades-old standard that tracked whether a learner completed a course and which score they received, but little else. xAPI, the Experience API standard that succeeded it, genuinely expanded what’s possible. It can capture nearly any digitally-observable learning event: simulations, mobile interactions, video behavior, off-LMS activity, performance support access, even real-world tasks when intentionally instrumented.
But xAPI captures activity, not learning itself. The statements it generates are evidence that something happened—but not always proof of learning transfer. Without a data strategy behind the technical infrastructure, a Learning Record Store—the database built to hold xAPI—fills rapidly with sophisticated noise. Collecting data and knowing what to ask of it are two entirely different skills.
A tool that captures everything tells you nothing if you never decided what you were looking for at the start.
When the Data We Trust Misleads Us
Here’s where it gets uncomfortable—because the instinctive data points many organizations rely on might be working against them.
In a landmark 2019 study published in the Proceedings of the National Academy of Sciences, Harvard researchers found that students overwhelmingly felt they learned more from traditional lectures, but actually scored significantly higher on tests following active learning sessions. As lead researcher Louis Deslauriers put it: “A superstar lecturer can explain things in such a way as to make students feel like they are learning more than they actually are.” The implication for Level 1 satisfaction surveys—the “smile sheets” that still anchor many organizations’ evaluation strategies—is direct and uncomfortable. A training that feels smooth, clear, and enjoyable doesn’t produce a reliable signal about whether learning occurred.
Research from Cornell University’s Department of Psychology compounds this. Kruger and Dunning found that people with limited knowledge or skill in a given area systematically overestimate their own competence, while those with genuine expertise tend to underestimate theirs. In a corporate training context, this creates a specific problem: the learner who most needs reinforcement is often the one most likely to walk away feeling prepared. Research examining self-assessment accuracy in clinical and pharmaceutical education is unambiguous: self-assessed confidence is not an accurate measure of competence, and self-reported perception should never serve as the primary measure of whether there was learning transfer.
A data point that feels reassuring and a data point that’s accurate are not always the same thing.
Capability vs. Conditions
Consider a representative who completes a clinical conversation training. They score well and leave feeling confident and prepared, and the dashboard reflects this—all green across the board. All boxes checked. Two weeks later, they’re in the field with a manager who’s never heard of the program, no reinforcement structure, and a product launch that’s already displaced the training from everyone’s priorities. The capability was built, but the conditions to use it were never put into play.
Learning builds capability, but capability is only one piece of what drives performance. In the scenario, the system recorded success, but the meaning of success never went beyond the completion screen.
Thomas Gilbert’s human performance model identifies the factors driving on-the-job performance as a combination of individual variables—knowledge, skill, confidence, attitude—and environmental ones: tools, resources, incentives, management support, clear expectations, organizational culture. L&D directly influences the individual factors, but the environment that learners return to is largely outside L&D’s hands.
Research by Robert Brinkerhoff found that only about 10-20% of training programs result in measurable job transfer. In many cases, the training works—and the environment doesn’t. The capability gets built, but the conditions for applying it never materialize. It’s a systems gap rather than a training design failure, and closing it requires collaboration across L&D, management, and organizational leadership that most measurement architectures aren’t designed to support.
What AI Can—and Cannot—See
AI is genuinely useful in this realm. It can scale sentiment analysis across open survey responses, surface patterns in LRS data that no human team could process manually, flag at-risk learners before they fail, and help correlate learning activity with downstream business metrics at speed and scale. Additionally, AI can help us grant additional learner support at the individual level through the use of chatbots and personalized learning.
While AI can read a dashboard, it cannot sit in a sales call. It cannot assess whether a life sciences professional’s response to a clinical objection reflects genuine knowledge transfer or a well-memorized script. It cannot tell you whether the confidence a learner self-reported after training was calibrated accurately enough to serve a customer—or a patient—well.
As AI becomes capable of predicting behavior change before it’s directly observable, the field faces a genuinely open question: does that shift our entire measurement paradigm from retrospective to prospective? If we can identify, before a learner returns to the field, whether they’re likely to transfer—and intervene earlier—what does that change about how we design training, and how we define accountability for outcomes?
We're Not in a Knowledge Economy Anymore
One of the most clarifying moments at DevLearn 2025 came from Dr. Keith Keating, whose book Hidden Value: How to Reveal the Impact of Organizational Learning has been on my reading list. His argument frames the current moment precisely: we’re no longer in a knowledge economy, or even a skills economy. We’re in a value economy. Knowledge alone isn’t enough. The differentiator is how tools, people, collaboration, and learning combine to create something the organization can act on.
That reframing changes what measurement is for. ROI is a calculation—useful, important, but a single output captured at a point in time. Value is a posture. It asks different questions: What changed because of this? Who can do something now that they couldn’t before? What organizational decision was better informed? In life sciences, value has a specific texture: it’s the clinical conversation that went differently, the formulary access objection that was navigated with genuine command, the patient who received the right therapy because the professional on the other side of the conversation was prepared.
So when xAPI hands you the ability to instrument nearly anything—when the sky is genuinely the limit—what do you focus on?
The answer that holds across every organization, every program, and every definition of business outcome: measure the causal chain.
- Knowledge gain — did something actually transfer, or did they just complete the module?
- Confidence calibration — not just whether confidence is present, but whether it’s accurate, because an overconfident learner is a different risk than a prepared one.
- Behavior change — the hardest link to close, requiring human observation, manager input, and an environment that supports application.
- Business result — the only thing leadership is ultimately measuring, and the place most measurement strategies start when it should be where they end.
Miss any link, and it becomes very tough to diagnose where the chain broke.
L&D professionals today have more measurement infrastructure than any generation before us. The question isn’t whether we can measure. It’s whether we’re designing deliberately enough and honestly enough.
At CMR Institute, we’re asking these questions and actively building toward answers. Evolving how we measure knowledge gain, confidence, and behavior change in life sciences learning is one of the most important initiatives we’re invested in right now. If your organization is on the same journey, let’s talk.
References
- Association for Talent Development (ATD). ATD History.
https://www.td.org/about-atd/history - Advanced Distributed Learning (ADL) Initiative. Experience API (xAPI).
https://adlnet.gov/projects/xapi/ - Brinkerhoff, Robert O. The Success Case Method: Find Out Quickly What’s Working and What’s Not. Berrett-Koehler Publishers.
- Deslauriers, Louis, McCarty, Logan S., Miller, Kelly, Callaghan, Kristina, & Kestin, Greg. “Measuring Actual Learning versus Feeling of Learning in Response to Being Actively Engaged in the Classroom.” Proceedings of the National Academy of Sciences (PNAS), 2019.
https://www.pnas.org/doi/10.1073/pnas.1821936116 - Deloitte. Leveraging Learning Analytics to Drive Business Impact.
https://www.deloitte.com/us/en/services/consulting/blogs/human-capital/learning-analytics-to-drive-business-impact.html - Gilbert, Thomas F. Human Competence: Engineering Worthy Performance. McGraw-Hill, 1978.
- Keating, Keith. Hidden Value: How to Reveal the Impact of Organizational Learning. Kogan Page, 2023.
- Kruger, Justin, & Dunning, David. “Unskilled and Unaware of It: How Difficulties in Recognizing One’s Own Incompetence Lead to Inflated Self-Assessments.” Journal of Personality and Social Psychology, 1999.
https://psycnet.apa.org/record/1999-15054-002 - Tyler Vigen. Spurious Correlations.
https://www.tylervigen.com/spurious-correlations - Princeton Metrics. Learning Measurement Workshop presented through LTEN. https://www.princetonmetrics.com/index.html
Shelly Cline
Shelly Cline is a Senior eLearning Developer & Instructional Designer at CMR Institute, with over a decade of experience creating engaging, learner-centered training programs. She specializes in translating complex concepts into accessible, impactful learning experiences and has led large-scale initiatives across industries. Shelly is particularly interested in the role of AI in learning and development, exploring how emerging technologies can enhance engagement and design efficiency. She is committed to continuous improvement and building inclusive learning experiences.
