Report: ML Model Monitoring

1. Status and Retraining Decision

The ML Model was monitored monthly over the evaluation year, July 2025 to June 2026. Catch share and alert rate are the primary rule triggers: when either breaches its limit in two consecutive months, the model is flagged for a full refit. Feature and score drift are secondary rule triggers, reported beside the verdict. The primary rule trigger fired in September and October 2025, on catch shares of 0.33 and 0.50 against a limit of 0.54, so the model is flagged for a full refit; the refit was not run within the evaluation year, and version 13 stayed champion throughout.

The retraining rule is evaluated every month: primary rules decide a refit, and secondary rules flag drift for investigation.

2. Monitoring Results

2.1 Performance

Catch share, the first of the two primary rule triggers, fell below the 0.54 limit in August, September and October, and again in December. Those three months held 10 of the year's 32 conditions and the ML Model caught 4; the 6 misses had no common cause: on 3 the score crossed the threshold without staying above it, 2 fell under an ALERT already open with no sign of the condition, and 1 came while a compressor point was re-centering after a repair. December's miss fell inside the ALERT that had held HMC-02's spindle since July. Every other month with conditions stayed above the limit.

Alert rate, the second primary rule trigger, left its band once, in August, at 0.40 times the reference: the ML Model opened 3 ALERTs against 11 in July, while episodes opened in July stayed open. Vendor Rules fell the same month, to 0.46 times its reference, with measured hours at their usual level, so August was a quiet month rather than a change in the model.

2.2 Drift

The secondary rule trigger flags drift when more than 30% of point-features (each point's 10 vibration measurements) or point scores (the ML Model's anomaly score on each point) sit more than 0.5 in scaled distance from their reference. Point scores drifted in all 12 months and point-features in 10. Drift calls for investigation, not a refit, so it did not change the verdict. The investigation found:

• The drift is in vibration level on the spindles: crest factor and kurtosis drifted in under 5% of point-months, and the compressors barely drifted.
• Narrow references inflate it: points referenced to a re-centering window alone drifted nearly twice as often as points on training references.
• Drift tracks real faults only loosely: point-features drifted twice as often in months with a confirmed condition open on the asset (56% against 27%), but two thirds of drifted point-months had no condition open, so drift alone is a weak sign of a fault.
• One point may show a developing fault: the HMC-07 flood pump drifted in every month, its running-speed energy at 2.5 times its level after the pump was replaced in November 2024, with no inspection since. It is worth an inspection.

2.3 Data Quality

A data-quality exception is a point-month with any implausible rows, or with more than 30% of its scheduled hours without a measurement. Exceptions ran at 19 to 29 point-months a month, 278 in all. Nothing points to a failing sensor.

The 8 compressor points were exceptions in every month (96): they take no measurement while idle or on standby, 46% of their scheduled hours, and their morning start-ups trip the implausible-row rule (1,965 of their 1,983 implausible rows are temperature jumps as the housing warms from idle: real readings that the jump rule flags). The other 182 are on the HMC points, mostly isolated implausible rows at 0.20% of measured hours, and the HMC point-months above 30% without a measurement are whole machines out of service (HMC-04 in May, HMC-08 in June).

2.4 Champion and Re-centering

Version 13 passed all five promotion gates once and was champion for the year, with a false-episode rate of 0.042 against a 0.411 limit. 32 points on 22 assets were re-centered 54 times after a repair or remount; the plausibility check flagged 5, on the HMC-01, HMC-03 and HMC-08 spindles.

3. Monitoring Log

The refit the rule calls for (training cutoff moved forward, healthy windows and thresholds re-derived, registered as a challenger for manual review) was not run within the evaluation year, which was held out to score one frozen model.

Appendix

Retraining rule

Acted-on alerts remove the failures that would label them, so recall in a live deployment reads lower than the condition record supports.

Drift references