Anomaly Detection for Infrastructure Sensors [/əˈnɑməli dɪˈtɛkʃən fər ˌɪnfrəˈstrəkʧər ˈsɛnsərz/] n - A data center monitoring system yields thousands of sensor readings each minute. Most are normal; a small fraction betoken genuine trouble. The labor is to part signal from noise without breeding so many false positives that operators cease to heed alerts, and without missing the slow faults that hide within the noise.

The Two Ways Detection Fails

Over-sensitive detection produces nuisance alarms; over-insensitive detection misses faults, and both are costly. Nuisance alarms bring alert fatigue, until operators dismiss the alert that mattered; missed faults bring unplanned outages. The threshold between them is no fixed number: it turns on the cost of a missed fault weighed against the cost of a false alarm. See Tuning alert thresholds for the operational side of this reckoning.

Formally the trade-off is described by the receiver operating characteristic, the curve tracing true positive rate against false positive rate as the detection threshold varies. The area beneath this curve is the primary metric of model quality.

Statistical Process Control

Shewhart control charts are the simplest of anomaly detectors. A reading is flagged when it falls outside $k$ standard deviations of its historical mean:

$$\text{Alert if }|y_t-\bar{y}|>k\hat{\sigma }$$

The standard setting $k=3$ gives a false positive rate of about 0.3% under Gaussian assumptions. In practice sensor readings are not Gaussian, and mean and variance drift with season, workload, and the age of the equipment. A static mean and standard deviation will yield systematic false positives during workload ramps, and systematic missed detections during gradual drift.

EWMA control charts adapt to slow shifts in the mean:

$$Z_t=\lambda y_t+(1-\lambda )Z_{t-1}$$

with alert limits $Z_{t-1}\pm L\hat{\sigma }\sqrt{\frac{\lambda }{2-\lambda }}$. This is the same EWMA of EWMA and RiskMetrics, turned from a financial return upon a sensor reading. The control limits are tighter than the Shewhart equivalent, and detect small sustained shifts the sooner.

Regression Residuals for Condition Monitoring

Many anomalies appear not in the absolute level of a sensor but in its departure from what physics predicts. A compressor discharge temperature 5 degrees above its expected value, given ambient and load, is anomalous; the same reading on a cold day under low load is innocent.

The approach is to fit a regression model:

$${\hat{y}}_t=f(x_t)$$

where $x_t$ is the vector of relevant covariates (ambient temperature, load, pressure), and then to watch the residual $e_t=y_t-{\hat{y}}_t$ with an EWMA chart. Thus the systematic variation is stripped away, and the anomaly signal is gathered into the residual.

See Predictive Maintenance for CRAC and CRAH Units and Predictive Generator Maintenance for particular applications of this approach.

Isolation Forests for Multivariate Anomalies

Where several sensors together make a system, an anomaly in one may become visible only when weighed beside the others. A chiller drawing more power than expected for its cooling output is anomalous, though neither power nor cooling load stands alone out of range.

The isolation forest algorithm raises an ensemble of random decision trees that isolate observations. Points in sparse regions of the feature space are isolated in fewer splits; these are the anomalies. The anomaly score is:

$$s(x,n)=2^{-\frac{E[h(x)]}{c(n)}}$$

where $E[h(x)]$ is the expected path length to isolate $x$, and $c(n)$ the average path length for a random dataset of size $n$. Scores near one are anomalies; scores near 0.5 are normal.

Isolation forests ask no distributional assumptions, handle mixed-type features, and scale to large sensor arrays. Their chief danger is that they are trained on history: whatever resembles the past is held normal, even a slow drift toward failure that ran through the whole of the training period.

Of Drift and Missing Readings

Sensors drift. Temperature probes gather calibration error; current transformers suffer the degradation of their insulation. Readings go missing during network outages or sensor replacements. Each of these contrives spurious anomalies or masks true ones.

A practical course: flag readings as suspect rather than anomalous when they differ from neighboring sensors in ways consonant with calibration drift rather than physical change. Cross-validate each sensor against a spatial or physical model, and when a sensor gainsays the model, flag it; do not yet alarm the operator.

The Kalman filter, as set down in State Space Models and the Kalman Filter, takes missing data in its stride: when a reading is unavailable the update step is skipped and the prediction serves alone. Thus a sensor outage is kept from breeding a cascade of spurious anomalies through the monitoring system.