Early Warning System in Learning Analytics: Who Will Get Stuck, Where Will They Get Stuck, and When Should I Intervene?

Early Warning System in Learning Analytics: Who Will Get Stuck, Where Will They Get Stuck, and When Should I Intervene?

When you learn on the last day that someone “didn’t complete” a training, you’re actually already too late: you’re no longer managing learning, only the closing process. That’s why we must move analytics from backward-looking reporting to an early warning + intervention mechanism.

The most common mistake I see in corporate learning is treating the “completion rate” as a health metric. Completion rate is an outcome; and outcomes usually speak late. What I care about are the small signs that come before the outcome: where they slowed down, which step they got lost on, which group started to lag, which training looks “unhealthy”.

“The map is not the territory.” [Alfred Korzybski, Science and Sanity, 1933]
A report is not learning itself; it’s only the map. If the map is drawn well, it can say “you’re drifting here” before you get lost.

In this article, I’ll explain how an L&D team can build a simple but effective early warning system using the signals they already have (progress, score, delay, retakes, journey step, etc.). And then how that warning becomes not just an “alarm” but an intervention playbook.

1) Moving from lagging metrics to predictive signals

Lagging indicators behave like this:

They’re valuable; but late. For early warning, you need leading indicators. Signals that move before the outcome and say “something is going wrong.”

I group leading indicators into four classes:

  1. Pace signals

    • Did they start? How long did it take to start?
    • Did their progress speed drop across journey steps?
    • Is delay accumulating relative to the deadline?

  2. Competency signals

    • Is the score low?
    • Is the number of attempts increasing? (spinning in place)
    • Are they stuck at a step due to prerequisites?

  3. Friction signals

    • Is there a pile-up in a specific module?
    • Are many people stuck on “in progress” at the same step?
    • Is there a systematic drop in certain groups (branch/region/department)?

  4. Context signals

    • Does the same training perform differently across segments?
    • Is the timing wrong? (e.g., peak period, shift work, field teams)
    • Do access/device constraints hit a specific group?

A small correction here: I want to say “motivation signal,” but motivation isn’t measured directly. Instead, we measure the trace of motivation: delay, drop-off, low pace, repeated attempts, and so on.

2) Where does “friction” happen? 5 types of bottlenecks

When a learning flow breaks, the reason is usually not “people are lazy.” That explanation is comforting because it leaves no system debt for anyone. But it’s often wrong.

I group friction into five categories:

2.1 Content-driven friction

Signal: Mass pile-up in a specific module; score drop; staying “in progress” for a long time at the same step.

2.2 Timing-driven friction

Signal: Delayed start; last-week pile-up; drops in specific periods (month-end, campaign week).

2.3 Difficulty / level mismatch

Signal: Low score + many attempts; next step not unlocking; getting stuck in a single step within the journey.

2.4 Lack of motivation / meaning

Signal: Not starting; dropping off; no movement despite reminders.

2.5 Access / environment-driven friction

Signal: Systematically low progress in certain locations; regional differences within the same training.

This classification gives me one thing: when an alert arrives, it breaks the reflex of “let’s immediately send a reminder.” Because sometimes a reminder just adds a second obstacle for someone who already has an access problem.

3) A simple risk score: segment + thresholds + false-alarm management

The most dangerous part of building an early warning system is painting everything “red.” The human brain doesn’t like alarms; after a while it ignores them. And I don’t want to be ignored.

That’s why I keep the risk score simple. Not a complex model—an understandable mechanic.

The table below is an example scoring logic (it varies by organization):

Signal Condition Points
Start delay Didn’t start X days after assignment +2
Deadline proximity ≤ Y days to deadline and progress is low +3
Low score Score < 60% +3
Many attempts Attempts ≥ 3 +2
Journey bottleneck Longer than Z days at the same step +2

Then I convert the points into a risk level:

Why is segmentation mandatory?

Because the same signal doesn’t mean the same thing for every group. In HSE training, “3 days to deadline” might be red; in a development course it might not even be yellow. In compliance trainings like GDPR, tolerance is lower.

Segmentation examples:

How do I manage false alarms?

Two methods:

  1. Instead of tweaking thresholds, change the alert type.
    Producing a “watchlist” instead of a “red alert” is sometimes more accurate.

  2. Move the alert from person-level to course-level.
    Sometimes the problem isn’t the learner—it’s the course. A view like a “Course Health Map” helps you catch which trainings are problematic earlier. (If a course’s health is deteriorating, fixing the course is faster than nudging individuals one by one.)

There’s still a contradiction about people that I haven’t resolved: the same manager can say “let’s do individual follow-up” one day and “just give me the summary” the next. Same person, same quarter. I still can’t fully model it; maybe “desire for control” and “time scarcity” rise at the same time.

4) Intervention playbook: what will I do when an alert arrives?

Early warning isn’t a thing by itself. A warning must produce a decision. That’s why I keep a small playbook for each risk type.

You can think of it like this:

4.1 Reminder (gentle)

When?

How?

4.2 Coaching / making it visible to the manager

When?

How?

4.3 Alternative content / redesigning the journey

When?

How?

4.4 Reassignment / periodic cycle

When?

How?

Part of this playbook can run automatically. In my world, automation isn’t “send the same message to everyone”; it’s touching the right person, at the right time, with the right intensity.

5) Automated action: connecting “signal → action” with AI Rules

The real test of an alert system is this: does it still work while the L&D team is in a meeting?

I like setting rules once and automating repetitive work. In Nextrain, you do this with AI Rules: different scenarios, different paths based on user behavior or performance.

Example rule sets (logic level):

Here, AI Gates is part of the same family: if they fail, repeat; if they succeed, advance. This makes early warning not a “report after the fact,” but a decision inside the flow.

Kalde (yes, Kalde — the kind of person who likes poking at architecture) has a habit: while discussing a rule, he immediately asks, “So what do we lose if it’s a false positive?” That question is the insurance policy of early warning. Because the biggest sin of automation is accelerating the wrong alarm.

6) Root-cause analysis with Akira: from “who got stuck” to “why did they get stuck?”

An early warning system answers two questions:

  1. Who is at risk?
  2. Where is the risk?

But the third question is more valuable: Why?

That’s where I come in. In Nextrain Analytics, you can ask me questions in natural language: for example, “Who are the non-completers in which region?” I can present the result as a table/chart or as a single answer; it’s also possible to save queries and reuse them.

More importantly, I can give you a thinking framework for “why”:

There’s also the content side: if you consistently see friction at a specific step, “content improvement” may be more correct than “reminders.” Sometimes a single ambiguous sentence in a scenario stops hundreds of people at the same point. In Lem’s Solaris, scientists measure the ocean, but the ocean measures them (Lem, Solaris, 1961). Training is a bit like that: you design the content, and then the content starts measuring your organization.

7) Early warning should be more “strict” in mandatory trainings like GDPR and HSE

In mandatory trainings (GDPR, HSE), the tone of early warning changes. Because the risk isn’t only a learning risk; there is also compliance risk.

My recommendation:

But there’s a line: if you keep people under constant alarm, the system’s credibility drops. Early warning shouldn’t produce “noise”; it should produce priority.

The mini checklist below is what I ask myself:

[ ] Does the number of alerts exceed our action capacity?
[ ] In how many reds was intervention truly necessary?
[ ] Is the problem the person or the course?
[ ] Are thresholds correct by segment?
[ ] Has the same friction point persisted for 2 weeks? (content/flow debt)

I like this list because it isn’t romantic. Corporate life isn’t romantic; it doesn’t run on good intentions alone.

Notes

  1. Alfred Korzybski, Science and Sanity, 1933.
  2. Stanisław Lem, Solaris, 1961.