MLOps Monitoring Report: Demand Forecasting

The demand forecasting model has been monitored monthly across its live window, January to June 2026. Forecast error and bias are the primary model triggers, while target, prediction and feature drift are leading indicators. Business KPIs including fill rates, stockout days and jobs held are also monitored against target levels.

A forecast can only be scored once its lead-time window has closed, so each month is judged on accuracy once at least 80% of its forecasts have matured. By June 30th that covers January to April; May and June are shown but not yet judged on accuracy.

The flag reads RETRAIN RECOMMENDED, meaning the model should be fully retrained on data through June. This is because, after bias correction, the forecasts for intermittent items ran high in March and April (+27% and +24%), outside the ±10% Retrain threshold: the correction factors, fit on the prior year's errors, over-correct intermittent items. Separately, the safety buffers should be recalibrated, because the fill rate achieved ran more than 1 point below target for a criticality group in January, February, March and May.

The monitoring rules are presented below across three tiers and evaluated every month. Model rules determine when retraining is needed, Secondary rules manage leading indicators that warrant investigation, and Policy rules set business KPI targets.

The same measures are also profiled by the platform's data profiling on the reorder schedule, the feature frame and the forecast-error table, with the thresholds above kept as the rule.

Forecast error for each complete month of data is presented below against the 46.5% Investigate threshold (the model's bias-corrected WAPE on 2025 data). From January to April, overall error stays below the reference (42.5% to 45.6%) and overall bias stays within 6%, meaning the model as a whole has not degraded.

Overall Forecast Error (WAPE) by Month

Overall Forecast Bias by Month

Examining by demand pattern, bias for intermittent items exceeded the Retrain threshold in March and April. This is the primary trigger behind this report's retraining recommendation. Forecast error stays below the Retrain threshold for every pattern, though smooth items in January and April, erratic items in March and intermittent items in March ran above their 2025 level (Investigate).

Forecast Error by Demand Pattern, Points Above 2025 Level

Forecast Bias by Demand Pattern, % Above or Below Actual

Drift compares each month's data with a fixed reference calculated using the Jensen-Shannon distance, flagged at 0.10. This calculation checks for statistical similarity: whether this month's values are similar to the reference data (the training rows for usage and features, and the held-out 2025 forecasts for the model's forecasts), where 0 means the two are identical and 1 means they do not overlap at all. It is checked three ways: on the targets (usage over the lead-time window), on the model's forecasts, and on the model's features.

Target drift. Distance between each month's actual usage over the lead-time window and the training rows. Target drift stays well under the threshold (0.026 to 0.054), meaning monthly usage over the lead-time window resembles the usage the model was trained on.

Target Drift Distance by Month

Prediction drift. Distance between each month's forecasts and the forecasts from the held-out 2025 year. Prediction drift stays under the threshold every month (at most 0.054), meaning the model is producing forecasts on the same scale and spread as in 2025.

Prediction Drift Distance by Month

Forecast Distribution: Held-out 2025 Reference vs Jun 2026

Feature drift. Per-feature distance between each month's inputs and the training rows, for the 24 features other than the calendar ones, which change with the date by design. Between 0 and 2 of the 24 features cross the threshold in any month, within the rule's allowance of 3, meaning the model's monthly inputs resemble the inputs that the model was trained on.

Per-Feature Drift Distance (feature by month)

Checks on each month's inputs: the forecasts made and scored, rows with a missing feature, and scored items with no rows in the training data. The inputs arrive complete every month: about 3% of rows carry a missing feature, and two scored items have no training rows.

Two checks read the stock records themselves. A short week is an item-week in which the ledger balance reaches zero; an active short week is one in which the part also had usage, a held job or a service backorder, so recorded usage may understate demand. On the 47 parts whose receipts were booked in a purchase unit, the ledger as booked carries $12.1K of stock against $3.2K on the physical count: receipts booked in purchase units inflate the book until the next count corrects them.

Over the past six months, fill rate measured below target in 11 of 18 months across the three item types, and by more than 1 point for a criticality group in January, February, March and May, the rule that calls for the safety buffers to be recalibrated on the January to June errors.

Fill Rate by Item Type and Month

Stockout days sit above the 2025 monthly average in January and below it from February onward; held jobs stay below the 2025 average every month.

Stockout Days and Held Jobs by Month

The review compares each part's pooled 26-week lead time with its corrected master lead time at each month-end, and flags a part when the pooled figure runs more than 25% above the master for 2 month-ends running; suppliers are checked the same way. A flagged master goes to purchasing for review; the reorder point keeps using it until it is changed. Parts flagged rose from 22 in January to 69 in June, with 2 supplier flags over the six months.

Parts Flagged for Lead-Time Review by Month

The model is refit at the start of each month on every forecast date whose outcome is known, with its tuned hyperparameters held fixed, and its bias-correction factors are re-estimated from the trailing 52 weeks of out-of-sample errors. The log below records each refit, for traceability.

Planned actions from this review: a full re-tune on data through June 2026, with the bias correction for intermittent items reviewed; recalibration of the safety-buffer multiples on the January to June errors; a review of the masters flagged by the lead-time review; and a check at the end of July of the May and June forecasts once they have matured.