ML Model Technical Overview: Demand Forecasting

The demand forecasting model learns from the shop's weekly usage of its stocked items. It scores 1,336 of the 1,339 stocked items; the other three are successor parts with no usage history yet. Each training row is one item on one forecast date. Its features describe the item's usage and order book up to that date, and its target is the usage over the following lead time, rounded to whole weeks.

The historical data is split by date and never shuffled. Training rows start at week 52 of the history, because every forecast needs a full year of usage behind it. A 26-week validation block before 2025 was used to tune and compare the candidates, and 2025 was held out to score the winner. Each split keeps its whole forecast horizon before the next window starts, so no training target overlaps a validation week and no validation target overlaps 2025.

The first six months of 2026 are the model's live output; the model is retrained monthly on every forecast date whose outcome is known.

Weekly Usage by Split

The model's 26 features fall into four groups. The first is Rolling Usage, including usage windows (4, 13, 26 and 52 weeks), the count of weeks with any usage, the time since the last usage and the recent variability. The second is Calendar Features, to let the model learn seasonal demand patterns. The third is Item Attributes, including cost, lead time, demand pattern, value class and category, to tell the model how to read each item's usage history. The fourth is the Order Book: the component requirements of jobs booked on or before the forecast date, namely the requirements due inside the lead-time window and in the four weeks after it, the subset already released, the open job count, the book's cover of average usage, and the product-family backlog. The correlation figures presented are Spearman rank correlations with the target on the training rows, which suit the skewed, zero-heavy usage.

The rolling usage windows are strongly correlated with one another, since they overlap in time. The weeks-since-last-usage and variability features move against the usage windows, as expected for intermittent items. Unit cost runs against usage (-0.44 to -0.48), because expensive items are used in smaller quantities. Last year's usage correlates only +0.71 with the 52-week total, so its timing adds information beyond the average level. The order-book features correlate weakly with the target because the target is usage net of booked requirements; their value is in the booked quantity itself, which enters the forecast directly, and in book_cover, which the model uses to read how much of an item's usage the book already explains.

Feature Correlation Heatmap (numeric features, Spearman)

The target is heavily right-skewed: most lead-time windows use a few dozen units, while fasteners and consumables run to thousands. The model is therefore trained on log(1 + usage), so the few very high-volume items do not dominate the fit and the model learns proportional patterns across items of very different sizes. This distribution is stable across the three splits, as shown below.

Target Distribution (usage net of booked requirements): Train / Validation / Held-out Test

Part of an item's usage over its lead time is known on the forecast date: the component requirements of jobs already booked. On the parts that sit on a bill of materials, those requirements cover 36% of usage over the lead time; parts on no bill have none. The order book's own timing error against what the floor actually backflushed in the window is 16.7% (WAPE). The model forecasts only the remainder, because the ERP nets the booked requirements itself: forecasting them too would count them twice.

Three candidates, a ridge regression, a random forest and an XGBoost gradient-boosted model, were each tuned with Optuna (12 trials per model) on the validation weeks, using weighted absolute percentage error (WAPE) on total lead-time demand (the booked requirements plus the forecast remainder) as the objective. The selection rule picks the lowest validation WAPE on total demand; the booster is preferred only when it finishes within half a point of the best, and within 0.3 points the smaller feature set wins. The models were then refit on train plus validation and scored once on the held-out 2025 year. The random forest had the lowest validation error and was selected.

The best random forest model has 300 trees with a maximum depth of 16 and at least 8 rows per leaf. The relatively large leaf-size floor keeps the model from fitting individual spikes and instead learning the items' typical usage patterns.

Accuracy is measured as WAPE: the total absolute gap between forecast and actual usage across all forecasts, divided by total actual usage. The fair comparison is the order book plus the best simple method by demand pattern, which knows the same booked requirements: on the held-out 2025 year it scores 47.2%, against 43.0% for the model's raw forecast. The reorder points use the bias-corrected forecast (Section 5), which trades some accuracy (43.0% to 46.5% on the held-out year) for forecasts that do not run systematically low. Live figures are the monitor's latest matured month.

A live WAPE of 45% means that for every 100 units an item used over its lead time, the forecast was off by about 45 units in either direction. Across the matured months from January to April it ran 42.5% to 45.6%. That is typical for item-level forecasts of lumpy, job-driven demand, where a single job or spare-parts order can double an item's usage in a week. The errors largely cancel across items: over all live forecasts scored by June 30, total forecast usage was within 1.3% of actual. Most of the operational improvement comes from how the forecast is used, covered in Section 5.

The learning curve refits the selected model on growing random shares of the training and validation rows. The held-out error flattens well before the full training set, so more of the same history adds little.

Learning Curve

Calibration compares the mean forecast with the mean actual usage within each tenth of the forecasts, from smallest to largest. Points on the diagonal mean a forecast of, say, 50 units really does average about 50 units of usage. After the bias correction described in Section 5, the live forecasts track actual usage within 10% from about 23 units up, with the largest tenth running 6% high. Below that, actual usage runs above the forecast: the smallest tenth of live forecasts average 1.1 units against 2.4 used, and the held-out year shows the same shape more strongly. The safety buffer absorbs that gap on the smallest items.

Calibration: Mean Forecast vs Mean Actual, by Forecast Decile

Error varies by demand pattern. Smooth items are forecast best and intermittent items worst, since an item used a few times a year at random offers little to learn from. The table compares the model with the order book plus the best simple method for each pattern, in the held-out year and live. The model's gain over the order book plus the best simple method is concentrated in lumpy and intermittent items in the held-out year; on smooth items the simple method is as good.

Live WAPE by Demand Pattern, Model vs Order Book + Best Simple Method

Plotted against actual usage on log scales, the live forecasts cluster around the diagonal across four orders of magnitude, meaning the same model tracks items that use a handful of units per lead time as well as those that use thousands. The spread around the diagonal is the item-level error that the safety buffer absorbs.

Forecast vs Actual Usage, Live 1H 2026

Before go-live, groups of candidate features were added one at a time to the usage-history set: the order book, product families, shortage history, item lifecycle (also on a longer training history), supply, and calendar effects. Each rung was tuned and scored the same way, on total demand over the lead time. Only the order book improved held-out accuracy on the raw forecast, and it had the lowest validation error (45.3%); the later groups added features without improving the held-out raw forecast, so the smaller set was kept. The per-rung tables are in the appendix.

Held-Out 2025 WAPE by Rung of the Feature Study

SHAP values measure each feature's average contribution to a forecast, here on a sample of the held-out 2025 forecasts and in units of log usage. The model leans most on usage in the same weeks last year, supplier lead time and the order book's cover of average usage. Last year's usage over the same weeks is by far the strongest, because it is the only feature measured over exactly the window being forecast. Lead time comes second, since it sets how long a window the forecast covers. The order book's cover of average usage comes third; the other order-book inputs, beyond the booked quantity itself, add little.

Calendar and category features contribute little. About 30% of items are seasonal, but each peaks at its own time of year, and last year's same-weeks usage already carries each item's own seasonal timing; a shared week-of-year or month feature adds little on top of it. Category adds little for a similar reason, since cost, usage level and demand pattern already capture most of what distinguishes the categories.

Mean Absolute SHAP Value by Feature

The model makes one prediction: the usage forecast, how many units each item will use over its supplier lead time. The reorder point and order quantity presented in the ERP are then calculated from that usage forecast using the rules below.

Bias correction. Trained on log usage, the model's back-transformed forecasts estimate the median rather than the mean, so they run low. Each demand pattern's forecasts are scaled by a factor re-estimated at each monthly retrain from the trailing 52 weeks of out-of-sample errors. The table shows the factors at the first live retrain and the corrected bias by month from the monitor. The correction removes the systematic under-forecasting for smooth, erratic and lumpy items; intermittent items were over-corrected in March and April.

Order quantity. The recommended order quantity for each item, calculated from its usage forecast, balances the cost of placing an order line against the cost of holding stock. Expensive, heavily used items are ordered every few weeks, and cheap items a few times a year.

The chart below groups items by value class. The classes are set by annual usage value (units used × unit cost): A items are those that together make up the first 80% of usage value, B items the next 15% and C items the last 5%. In 2025 the typical A item used about $11.5K of stock a year (most above $5.4K) at a median unit cost of $36; a typical B item about $3.1K a year at $11; and a typical C item about $640 a year at $2. The A items, where most of the money sits, are bought in the smallest lots. See the ML Model Overview report for more detail on this order quantity approach.

Order Quantity in Days of Usage, by Value Class

Safety buffer. The buffer is calculated as k standard errors of the item's own forecast error: k × σ, where σ combines the forecast's miss over the lead time with the variability of the supplier's delivery time, and k sets how many of those standard errors to hold. k is not taken from a normal curve. It is set on the model's actual errors so that the expected shortfall per replenishment cycle stays within the fill-rate target for the item's criticality group. The errors have much fatter tails than a normal curve: at k = 2 the expected shortfall is 2.5 times what a normal distribution implies, and at k = 3, 7.1 times. This is important to ensure an adequate buffer size. Because the shortfall allowance scales with the order quantity, an item ordered in large lots needs less buffer (its own lot protects most of the cycle), and an item ordered often needs more.

Expected Shortfall vs Buffer Size: Actual Errors vs Normal

• Irreducible item-level error. Lumpy and intermittent demand carries randomness no model can learn, and about two-fifths of items (507 of 1,339) are lumpy or intermittent; much of the item-level WAPE of 45% reflects that rather than the fit.
• Accuracy and operating result are not the same measure. The order book's gain in forecast accuracy did not move the forward results by as much, because the ERP already netted released jobs before the model saw them.
• Item-level lead times are not observable at low receipt counts. At 2 to 5 receipts a year, a 26-week statistic cannot track a drift in an item's lead time; supplier-level pooling can, and is used for review rather than for the reorder point.
• Bias on intermittent items in live use. After bias correction, the forecasts for intermittent items ran high in March and April, which the monitor flagged for retraining; the buffers on those items were larger than needed in those months.
• Superseded parts. The policy sets reorder points on superseded parts until their stock is used up; the ERP suppresses the order.
• Fill rate, not stockouts. The buffer targets the share of units supplied. It does not directly target the probability of a stockout or of a job being held, though these important metrics are closely monitored.
• Supply is taken as given. Lead times are refreshed from recent receipts and their variability enters the buffer, but late deliveries are not forecast.
• Simulated data. The model is trained and evaluated on generated ERP data; accuracy on a shop's own records depends on the signal in them.

How the model runs in production and where its inputs come from.

Getting the numbers into the ERP takes two planning fields per item per week, the reorder point and the order quantity. They are written through the ERP's API where it offers one, and otherwise through its import utility after the purchasing manager reviews the queue; the buyers review the queue in both cases.

Every reported figure comes from a fixed-seed run, archived with a checksum manifest and reproduced on the platform.

Held-out 2025 WAPE on total demand for the selected model at each rung, raw / bias-corrected.

Bias-correction factors by demand pattern for the selected model at each rung, estimated on validation.

Simple methods on the feature-study rows.

Leakage tests: the order-book inputs replaced by their values four weeks earlier. A forecast that leaned on information not yet available on the forecast date would lose much of its accuracy when lagged; these lose 0.2 to 1.3 points.