Two years of daily KPIs, a log of the breaks that really happened, and a calendar of the ones that only look like breaks. Build the alerting service before you read the walkthrough.
The situation
Ellerby Retail watches 6 numbers every morning. Conversion dropped last Tuesday and nobody could say whether it was real, whether it was the weekend effect, or whether it was noise. They want an alerting service.
You have 730 days of history from 2024-07-01 to 2026-06-30.
The data
| File | What it holds |
|---|---|
| ellerby-daily-kpis.csv | date, sessions, conversion_rate, aov, revenue, add_to_cart_rate, checkout_errors |
| ellerby-calendar.csv | Black Fridays, January sales and Christmas shutdowns. Use this one freely, it is operational knowledge rather than an answer |
| ellerby-incident-log.csv | The genuine breaks, with metric, dates, size and cause. This is the marking scheme. Do not open it until you have a detector |
What the room believes
- A three sigma rule on each metric is a reasonable starting point.
- A sequence model will beat a statistical baseline.
- The hard part is detection.
- A detector that catches more incidents is a better detector.
Definition of done
- A verdict on each of the four beliefs, with the evidence.
- At least three detectors compared on terms that are actually comparable, and an explanation of what makes them comparable.
- A false alarm rate you would be willing to put in front of an on-call rota, and the detection rate that buys.
- A statement of how quickly each incident was caught, not just whether it was.
- An honest list of the breaks your service will not catch, and what would be needed to catch them.
Five questions worth sitting with before you build anything
If one incident lasts sixty days, how many alerts should it produce, and how does that change how you score a detector? What is your baseline made of, and could an ongoing incident have got into it? How would you compare two detectors that fire at different rates? How many false alarms a month before somebody mutes the channel? And would your detector notice a metric that fell four per cent and stayed there?
If you want to go further
- Score your detector per incident rather than per day and see how different the picture looks.
- Fix a false alarm budget, tune every method to it, and only then compare what they catch.
- Separate your results by the kind of incident. Spikes and sustained shifts will not behave the same way.
- Try removing the day of week effect and measure what that alone is worth.
- Work out how many days late each detection was, and whether that would have mattered.
When you are done, read the walkthrough. It scores four detectors per incident, holds them all to the same false alarm budget, and finds that what separates them is not what they catch. Compare its budget to yours, and its list of misses to yours.
Six daily KPIs, two years, 11 real breaks and 8 calendar events that must not wake anyone. Held to the same alert budget, every method catches the same 8 of 11. What separates them is how much noise they make getting there, and the deep model is not the one that wins.
The situation. Ellerby Retail watches 6 numbers every morning. Conversion dropped last Tuesday and nobody could say whether it was real. 730 days of history, 11 genuine incidents with a known cause, and 8 Black Fridays and January sales that look exactly like incidents and are not.
What the business is left with. An alerting service with a stated false alarm budget, the detection rate that budget buys, and an honest list of the breaks it will never catch.
Attempt it first. The brief has the same series, the calendar, and an incident log to score against.
Contents
Section 1Problem Definition
No code yet. Anomaly detection on business metrics is usually framed as a modelling problem and is almost entirely an alerting problem, and the two have different success criteria.
The problem in one sentence
A detector that catches everything wakes somebody up twice a week for nothing, and a detector nobody reads catches nothing at all.
| What moves a daily KPI | How often | Should it alert |
|---|---|---|
| Day of week | Every week | Never. revenue swings 45.0% between its best and worst day |
| A calendar event | 8 times in two years | Never. Black Friday is not an incident |
| Ordinary noise | Constantly | Never, and some of it will clear three standard deviations on its own |
| A genuine break | 11 times in two years | Always, and quickly enough to matter |
Business objective
Alert on the 11 and stay quiet the rest of the time, inside a false alarm rate the on-call rota will actually tolerate.
Hypotheses
- H1. A three sigma rule on each metric is a reasonable starting point.
- H2. A sequence model will beat a statistical baseline.
- H3. The hard part is detection.
- H4. A detector that catches more incidents is a better detector.
Why the unit here is the incident, not the day
An outage that lasts sixty days is one thing to be told about, not sixty. Every method below is scored on whether it fired on the right metric within 7 days of a break starting, how long that took, and how many times it fired when nothing was wrong. Alerts are suppressed for 7 days after firing on a series, because that is what any real alerting service does.
Scoring day by day instead produces precision figures in the teens for every method and tells you nothing, because a rolling baseline absorbs a long incident and stops flagging it. That is a property of the metric, not of the detector.
Section 2Data Collection
import numpy as np
import pandas as pd
d = pd.read_csv('data/ellerby-daily-kpis.csv', parse_dates=['date'])
inc = pd.read_csv('data/ellerby-incident-log.csv', parse_dates=['start_date', 'end_date'])
cal = pd.read_csv('data/ellerby-calendar.csv', parse_dates=['start_date', 'end_date'])
KPIS = [c for c in d.columns if c != 'date']
d['dow'] = d['date'].dt.dayofweek
print('days :', len(d))
print('kpis :', len(KPIS))
print('window :', d['date'].min().date(), 'to', d['date'].max().date())
print('incidents :', len(inc), dict(inc['kind'].value_counts()))
print('calendar :', len(cal), 'events that are not incidents')days : 730
kpis : 6
window : 2024-07-01 to 2026-06-30
incidents : 11 {'spike': 5, 'shift': 4, 'tracking': 2}
calendar : 8 events that are not incidents| File | What it is |
|---|---|
| ellerby-daily-kpis.csv | One row per day. sessions, conversion_rate, aov, revenue, add_to_cart_rate, checkout_errors |
| ellerby-incident-log.csv | The 11 genuine breaks with their start, end, metric, kind and cause. This is the marking scheme |
| ellerby-calendar.csv | Black Fridays, January sales and Christmas shutdowns. Large, real, and not something anybody wants an alert about |
Three kinds of incident: 5 spikes that last a day or two, 4 level shifts that last weeks, and 2 tracking failures where the number drops because the measurement broke rather than the business. Section 7d is largely about the difference between the first two.
Section 3Data Preprocessing
3aDuplicates and schema checks
print('duplicate dates :', int(d['date'].duplicated().sum()))
print('missing days :', int((d['date'].diff().dt.days.dropna() != 1).sum()))
print('missing values :', int(d[KPIS].isna().sum().sum()))
print('revenue reconciles :',
bool(np.allclose(d['revenue'],
d['sessions'] * d['conversion_rate'] * d['aov'], rtol=0.35)))duplicate dates : 0
missing days : 0
missing values : 0
revenue reconciles : TrueNo gaps in the calendar, which matters more than usual: a missing day in a daily series either becomes a zero and alerts, or gets skipped and quietly shifts every window that follows it.
3bHandling categorical mess
Day of week is the only categorical field and it is the largest single source of movement in the file. It is treated as a feature to be removed rather than one to be modelled.
3cDealing with outliers
Nothing is trimmed, because the outliers are the thing being detected. The point of the project is telling which outliers matter.
3dHandling missing values
None present. On a real feed the common failure is a partial day at the end of the series, which reads as a catastrophic drop and is the single most common false alarm in production alerting.
3eHandling skewed data
Everything is analysed in logs. A percentage move means the same thing at every level of a series, which a raw difference does not, and revenue moves over a range wide enough for that to matter.
3fData types and normalisation
Baselines are trailing and stop a week short of the day being tested. That gap is load bearing: without it, a level shift walks into its own baseline and the detector concludes that the new level is normal within a fortnight.
Section 4Exploratory Data Analysis
4aTarget variable analysis
11 incidents across 173 series days, which is 4.0% of the 4,380 days in the file. Rare enough that a detector which never fires is right almost all the time, which is the same trap as any rare event problem.
4bNumerical variables
for k in KPIS:
by_dow = d.groupby('dow')[k].mean()
z = (d[k] - d[k].mean()) / d[k].std()
print('%-18s day of week swing %5.1f percent cv %.3f days beyond 3 sd %2d'
% (k, 100 * (by_dow.max() / by_dow.min() - 1), d[k].std() / d[k].mean(),
int((z.abs() > 3).sum())))sessions day of week swing 24.0 percent cv 0.190 days beyond 3 sd 16
conversion_rate day of week swing 8.8 percent cv 0.119 days beyond 3 sd 8
aov day of week swing 8.0 percent cv 0.076 days beyond 3 sd 6
revenue day of week swing 45.0 percent cv 0.292 days beyond 3 sd 8
add_to_cart_rate day of week swing 7.1 percent cv 0.086 days beyond 3 sd 4
checkout_errors day of week swing 15.1 percent cv 0.313 days beyond 3 sd 5revenue is 45.0% higher on its best weekday than its worst. That is larger than several of the incidents in the file. Any rule applied to the raw series spends most of its alerts rediscovering the week.
# Revenue is the product of the other three, so one break shows up in two places.
# Except on the days when something hit revenue itself, and that gap is a signal.
approx = d['sessions'] * d['conversion_rate'] * d['aov']
gap = (d['revenue'] - approx).abs() / d['revenue']
rev_inc = np.zeros(len(d), dtype=bool)
for r in inc[inc['kpi'] == 'revenue'].itertuples():
rev_inc |= ((d['date'] >= r.start_date) & (d['date'] <= r.end_date)).values
print('median gap to the identity : %.4f percent' % (100 * gap.median()))
print('largest gap, ordinary days : %.4f percent' % (100 * gap[~rev_inc].max()))
print('largest gap, revenue incident days: %.1f percent' % (100 * gap[rev_inc].max()))
print('correlation with sessions : %.3f' % d['revenue'].corr(d['sessions']))
print('correlation with conversion : %.3f' % d['revenue'].corr(d['conversion_rate']))median gap to the identity : 0.0095 percent
largest gap, ordinary days : 0.0305 percent
largest gap, revenue incident days: 138.1 percent
correlation with sessions : 0.909
correlation with conversion : 0.79The identity holds to 0.0305 per cent on ordinary days and breaks by up to 138.1 per cent on the 22 days when something hit revenue directly. A revenue move its own components cannot explain is a different kind of event from one they can, and a service that watched only the identity would find those days for free.
4cCategorical variables
The calendar is the second thing a naive detector finds. 8 events cover 20 days, and they are among the largest movements in the file. Section 7b counts how many alerts they cost.
4dRelationships between variables
Revenue is sessions times conversion times order value, so anything that hits one of the three shows up in it too. That is why an incident on conversion is counted against conversion and not also against revenue: one break should produce one alert, and a detector that fires on every correlated series is a detector nobody reads.
4eTesting our hypotheses
| Hypothesis | Verdict | Evidence |
|---|---|---|
| H1. Three sigma on each metric is a reasonable start | No | On the raw series it costs 21 alerts on calendar events alone and still misses 2 incidents |
| H2. A sequence model will beat a statistical baseline | No | At the same budget the autoencoder catches 8 of 11, the same as everything else, with 1.62 false alarms a month against 0.42 |
| H3. The hard part is detection | No | Every method at the budget catches the same 8. They differ by a factor of 4.2 in noise |
| H4. More incidents caught means a better detector | Only at equal noise | Detection rate rises with any threshold you loosen. It is meaningless without the false alarm rate beside it |
4fSubgroups
The subgroup that matters is the kind of incident. Spikes and level shifts need different tests, and section 7d is where that becomes unavoidable.
Section 5Feature Engineering
5aThe leakage trap
| The trap | What it looks like | What it does |
|---|---|---|
| A centred rolling baseline | A window that includes days after the one being tested | The incident is in its own baseline. A first pass at this project used a centred 56 day median and detected almost nothing, because a 60 day shift is invisible against a window that contains it |
| A trailing baseline with no gap | Yesterday through to 28 days ago | A level shift becomes the new normal within a fortnight and the alert stops. Every baseline here stops seven days short |
| Fitting the seasonality on the whole series | Day of week factors from all 730 days | The incidents contribute to the factors that are supposed to exclude them |
5bNew features
| Feature | How it is built | Why |
|---|---|---|
| Log level | Natural log of each metric | A percentage move reads the same at every level |
| Day of week factor | Median residual against a trailing 91 day baseline, per weekday | Removes the largest non incident source of movement |
| Trailing median and MAD | 28 day window ending 7 days before today | A robust baseline that an ongoing incident has not yet contaminated |
| Calendar flag | From the published event list | Known large movements that are not incidents |
5cEncoding
The robust scale is the median absolute deviation times 1.4826, not the standard deviation. On a series that contains occasional genuine spikes, a standard deviation is inflated by exactly the events it is supposed to help you find.
5dFeature selection
Nothing is selected away. The choice being made here is what to remove from the signal before testing it, and each removal is measured separately in section 7b so the contribution of each is visible.
Section 6Model Selection
| Method | What it assumes | What it costs |
|---|---|---|
| Trailing z score on the raw series | Today looks like the last four weeks | Nothing. Four lines |
| Day of week removed, then z score | Today looks like the last four weeks, once the weekday is accounted for | One extra step |
| The same, with calendar events suppressed | And that Black Friday is not news | A list somebody maintains |
| LSTM autoencoder on 14 day windows | Normal weeks reconstruct well and abnormal ones do not | 300 epochs, 24 hidden units, and a model to keep running |
How the autoencoder is trained, and why it matters
It is fitted on 299 windows that contain no incident at all, which is a considerable advantage handed to it: on a real series nobody knows which weeks were clean. If it cannot beat a z score under those conditions it will not beat one in production either.
Section 7Model Training
7aBaselines
DETECT_WINDOW, SUPPRESS = 7, 7
KPI_IX = {k: j for j, k in enumerate(KPIS)}
N, K = len(d), len(KPIS)
running = np.zeros((N, K), dtype=bool)
for r in inc.itertuples():
m = ((d['date'] >= r.start_date) & (d['date'] <= r.end_date)).values
running[m, KPI_IX[r.kpi]] = True
calm = np.zeros(N, dtype=bool)
for r in cal.itertuples():
calm[((d['date'] >= r.start_date) & (d['date'] <= r.end_date)).values] = True
onsets = [(int(np.where(d['date'] == r.start_date)[0][0]), KPI_IX[r.kpi], r)
for r in inc.itertuples()]
def suppress(raw):
# One alert per break, not one a day for the length of it.
out = np.zeros_like(raw, dtype=bool)
for j in range(K):
last = -99
for i in range(N):
if raw[i, j] and i - last >= SUPPRESS:
out[i, j] = True
last = i
return out
def evaluate(alerts, label, skip_calendar=False):
a = suppress(alerts)
if skip_calendar:
a = a & ~calm[:, None]
detected, delays = 0, []
claimed = np.zeros_like(a, dtype=bool)
for i0, j, r in onsets:
hit = np.where(a[i0:i0 + DETECT_WINDOW, j])[0]
if len(hit):
detected += 1
delays.append(int(hit[0]))
claimed[i0 + hit[0], j] = True
false = a & ~running & ~claimed
return {'method': label, 'detected': detected, 'of': len(onsets),
'median_delay': int(np.median(delays)) if delays else None,
'false_per_month': round(false.sum() / (N / 30.4), 2),
'calendar_false': int((false & calm[:, None]).sum())}LOG = np.log(np.clip(d[KPIS].values.astype(float), 1e-9, None))
def trailing(mat, gap=7, win=28):
# A baseline that stops a week short, so an incident cannot walk into it.
df = pd.DataFrame(mat)
med = df.shift(gap).rolling(win, min_periods=14).median()
mad = (df.shift(gap).rolling(win, min_periods=14)
.apply(lambda v: np.median(np.abs(v - np.median(v))), raw=True))
return med.values, mad.values
def zscore_alerts(mat, k):
med, mad = trailing(mat)
scale = np.where(mad > 0, mad * 1.4826, np.nan)
return np.nan_to_num(np.abs((mat - med) / scale)) > k
for k in (3.0, 4.0, 5.0):
r = evaluate(zscore_alerts(LOG, k), 'raw, k=%.0f' % k)
print('%-14s caught %2d/%d false/month %5.2f calendar false %2d'
% (r['method'], r['detected'], r['of'], r['false_per_month'],
r['calendar_false']))def dow_factors(mat):
out = np.zeros_like(mat)
df = pd.DataFrame(mat)
for j in range(mat.shape[1]):
base = df[j].shift(7).rolling(91, min_periods=28).median()
resid = pd.Series((df[j] - base).values, index=d['dow'].values)
out[:, j] = d['dow'].map(resid.groupby(level=0).median()).values
return np.nan_to_num(out)
DES = LOG - dow_factors(LOG)
for k in (3.0, 4.0, 5.0):
r = evaluate(zscore_alerts(DES, k), 'deseasonalised, k=%.0f' % k)
print('%-24s caught %2d/%d false/month %5.2f calendar false %2d'
% (r['method'], r['detected'], r['of'], r['false_per_month'],
r['calendar_false']))
r = evaluate(zscore_alerts(DES, 4.0), 'plus calendar', skip_calendar=True)
print('%-24s caught %2d/%d false/month %5.2f calendar false %2d'
% (r['method'], r['detected'], r['of'], r['false_per_month'], r['calendar_false']))raw, k=3 caught 9/11 false/month 2.71 calendar false 21
raw, k=4 caught 7/11 false/month 1.17 calendar false 17
raw, k=5 caught 7/11 false/month 0.92 calendar false 16A plain rule on the raw series spends most of its budget on the calendar: 21 of its false alarms at the loosest setting are Black Fridays and January sales.
deseasonalised, k=3 caught 5/11 false/month 3.62 calendar false 22
deseasonalised, k=4 caught 7/11 false/month 1.67 calendar false 18
deseasonalised, k=5 caught 8/11 false/month 1.08 calendar false 16
plus calendar caught 7/11 false/month 0.92 calendar false 0Every row above uses a different threshold and therefore makes a different amount of noise, which is exactly why they cannot be compared as they stand.
7bComparing candidates
Those rows are not comparable, because each setting produces a different amount of noise. A detector can always catch more by alerting more. The only fair comparison fixes the noise and asks what each method catches within it.
BUDGET = 2.0
def at_budget(make_alerts, params, label, skip_calendar=False):
grid = [(p, evaluate(make_alerts(p), label, skip_calendar)) for p in params]
ok = [g for g in grid if g[1]['false_per_month'] <= BUDGET]
pick = max(ok, key=lambda g: g[1]['detected']) if ok else grid[-1]
print('%-42s param %5.2f caught %2d/%d false/month %5.2f'
% (label, pick[0], pick[1]['detected'], pick[1]['of'],
pick[1]['false_per_month']))
return pick
ks = np.arange(2.0, 8.01, 0.25)
at_budget(lambda k: zscore_alerts(LOG, k), ks, 'Raw series')
at_budget(lambda k: zscore_alerts(DES, k), ks, 'Day of week removed')
at_budget(lambda k: zscore_alerts(DES, k), ks,
'Day of week removed, calendar suppressed', skip_calendar=True)Raw series param 3.50 caught 8/11 false alarms per month 1.75
Day of week removed param 5.00 caught 8/11 false alarms per month 1.08
Day of week removed, calendar suppressed param 5.00 caught 8/11 false alarms per month 0.42
LSTM autoencoder param 98.25 caught 8/11 false alarms per month 1.62This is the whole result. Every method catches the same 8 incidents. Removing the day of week cuts false alarms from 1.75 a month to 1.08. Suppressing the published calendar takes it to 0.42. The autoencoder, given clean training windows nobody would have in real life, manages 1.62.
Two lines of seasonality handling beat the sequence model by a factor of four
Raw to calendar suppressed is a 76 per cent reduction in false alarms at the same detection rate. Raw to autoencoder is 7 per cent.
The deep model is not bad. It is simply solving a problem that the day of week factor and a published calendar have already solved, more cheaply and in a form the person on call can argue with at seven in the morning.
import torch
import torch.nn as nn
W = 14
Xs = np.nan_to_num((DES - np.nanmean(DES, axis=0))
/ (np.nanstd(DES, axis=0) + 1e-9)).astype(np.float32)
wins = np.stack([Xs[i:i + W] for i in range(N - W + 1)])
quiet = np.array([not running[i:i + W].any() for i in range(N - W + 1)])
train = wins[quiet][:int(0.7 * quiet.sum())]
class AE(nn.Module):
def __init__(self, k, h=24):
super(AE, self).__init__()
self.enc = nn.LSTM(k, h, batch_first=True)
self.dec = nn.LSTM(h, h, batch_first=True)
self.out = nn.Linear(h, k)
def forward(self, x):
_, (hn, _) = self.enc(x)
y, _ = self.dec(hn[-1].unsqueeze(1).repeat(1, x.shape[1], 1))
return self.out(y)
torch.manual_seed(0)
ae = AE(K)
opt = torch.optim.Adam(ae.parameters(), lr=5e-3)
tt = torch.tensor(train)
for _ in range(300):
opt.zero_grad()
loss = nn.functional.mse_loss(ae(tt), tt)
loss.backward()
opt.step()
ae.eval()
with torch.no_grad():
rec = ae(torch.tensor(wins)).numpy()
ERR = np.zeros((N, K))
ERR[W - 1:] = np.abs(rec - wins)[:, -1, :]
at_budget(lambda q: ERR > np.percentile(ERR[W:], q, axis=0),
np.arange(90.0, 99.9, 0.25), 'LSTM autoencoder')LSTM autoencoder param 98.25 caught 8/11 false/month 1.627cHyperparameter tuning
There is one parameter and it is the alert budget. Everything else follows from it. At 12 minutes to triage an alert, 2.0 false alarms a month is about 4.8 hours a year of somebody checking something that was fine. Double the budget and you buy very little extra detection and a great deal of extra scepticism.
Read without the false alarm column beside it, this chart says the best detector is whichever one was set loosest. That is the mistake the alert budget exists to prevent, and it is why section 7b fixes the noise before comparing anything.
7dFinal evaluation
# Which breaks the chosen detector actually caught, and how quickly.
a = suppress(zscore_alerts(DES, 5.0)) & ~calm[:, None]
for i0, j, r in onsets:
hit = np.where(a[i0:i0 + DETECT_WINDOW, j])[0]
print('%-18s %-9s %-42s %s'
% (r.kpi, r.kind, r.cause,
('caught, day %d' % hit[0]) if len(hit) else 'missed'))sessions spike bot traffic from one referrer caught, day 0
sessions tracking analytics tag dropped on category pages caught, day 0
conversion_rate shift checkout redesign shipped missed
checkout_errors spike payment provider incident caught, day 0
revenue spike site outage for most of the day caught, day 0
aov shift free delivery threshold raised caught, day 1
checkout_errors spike certificate expiry caught, day 0
add_to_cart_rate shift stock feed degraded missed
sessions tracking consent banner misconfigured caught, day 0
conversion_rate spike discount code leaked then revoked missed
revenue shift paid search budget cut missedThe level shifts nobody caught are the more interesting result.
| Metric | Kind | Cause | Caught |
|---|---|---|---|
| sessions | spike | bot traffic from one referrer | yes |
| sessions | tracking | analytics tag dropped on category pages | yes |
| conversion rate | shift | checkout redesign shipped | no |
| checkout errors | spike | payment provider incident | yes |
| revenue | spike | site outage for most of the day | yes |
| aov | shift | free delivery threshold raised | yes |
| checkout errors | spike | certificate expiry | yes |
| add to cart rate | shift | stock feed degraded | no |
| sessions | tracking | consent banner misconfigured | yes |
| conversion rate | spike | discount code leaked then revoked | no |
| revenue | shift | paid search budget cut | no |
Spikes and tracking failures are caught almost every time. Level shifts are not: 1 of 4. A test that asks whether today is unusual cannot see a metric that moved once and stayed there, and no threshold changes that.
Point detection catches spikes. It does not catch drift
3 of the 4 level shifts were missed by every method here, and every spike but one was caught. A z score against a trailing baseline asks whether today is unusual. A conversion rate that falls four per cent and stays there never produces an unusual day.
That is not a tuning failure and no threshold fixes it. It needs a second test that compares a recent window against an older one rather than a day against a baseline, and section 8 hands that over as the next piece of work rather than pretending this one covered it.
Section 8Documentation and Handoff
What to do, and who owns it
| Action | Detail | Owner |
|---|---|---|
| Remove the day of week before testing anything | It is the largest non incident movement in the file, up to 45.0% on revenue, and removing it costs one line | Analytics |
| Suppress the published calendar | It took calendar false alarms from 21 to 0. Somebody has to maintain the list | Analytics |
| Set a false alarm budget before choosing a threshold | 2.0 a month here. Without one, every comparison is a comparison of how loosely each detector was set | Product and Analytics |
| Use a trailing baseline with a gap | Seven days. Without it a level shift becomes the new normal and the alert stops | Analytics |
| Add a changepoint test for drift | 3 of 4 level shifts were missed by every method here. Compare a recent window against an older one | Analytics |
| Route the alert with its context | The metric, the size of the move, and whether a calendar event is running. An alert that only says something changed will be ignored by the third week | Engineering |
What not to do
- Do not compare detectors on how much they catch. Every method here catches the same 8 once the noise is equalised.
- Do not run a three sigma rule on a raw daily series. It spends 21 alerts on the calendar before it finds anything.
- Do not use a centred window. The incident ends up inside its own baseline, which is how a first pass at this detected almost nothing.
- Do not use the standard deviation as the scale. The spikes you are hunting inflate it.
- Do not reach for a sequence model first. Given incident free training windows it still made 3.9 times the noise of a z score with the weekday removed.
- Do not claim to detect drift with a point test. Say which kinds of break the service covers and which it does not.
Reproducibility
| Item | Value |
|---|---|
| Files | ellerby-daily-kpis.csv, ellerby-incident-log.csv, ellerby-calendar.csv |
| Window | 2024-07-01 to 2026-06-30, 730 days |
| Scoring | Per incident. A hit is an alert on the right metric within 7 days of onset, with alerts suppressed for 7 days after firing |
| Baseline | Trailing 28 day median and MAD, ending 7 days before the day tested |
| Chosen method | Day of week removed, calendar suppressed, k = 5.0 |
| Budget | 2.0 false alarms per month |
| Autoencoder | LSTM, 14 day windows, 24 hidden units, 300 epochs, trained on 299 incident free windows |
| Libraries | pandas, numpy, torch |
What to take from this
- Score alerting on the incident, not on the day. A sixty day outage is one thing to be told about.
- Fix the noise before comparing detectors. Detection rate on its own ranks detectors by how loosely they were set.
- Seasonality handling is worth more than the model. Here it was worth four times as much.
- Trailing baselines need a gap. Otherwise the incident teaches the detector that the incident is normal.
- Spikes and drift are different problems. One test will not do both, and saying so is part of the handoff.
- An alert budget is a product decision. It belongs to whoever carries the pager, not to whoever built the model.
The question was whether last Tuesday’s drop was real. The service answers it for spikes and tracking failures inside a budget of 0.42 false alarms a month, and it does that with a median absolute deviation and a list of bank holidays. The deep model was given clean training data and still lost. The part still outstanding is drift, which is 3 of the 11 incidents in this file and needs a different test rather than a better one.
Generate the monitoring this project hands over
Monitoring and Drift Code Generator
The recommendation here ends with a changepoint test for the drift a point detector cannot see. This generates drift and monitoring code with thresholds calibrated against your own data rather than against a default nobody checked.
Free, no signup. Runs in the browser.
The architectures behind the model that lost
Deep Learning Cheatsheet
The autoencoder here is an LSTM encoder, a repeat, and an LSTM decoder. This is every architecture and training concept on one page, including the recurrent layers this project leans on and the reason they were the wrong tool.
Free, one page, LAD branded. No signup.
Companion projects. Anomaly Detection When Positives Are Rare is the same rare event problem on transactions rather than time series, sized to a review team. The Funnel That Leaks is the project about telling a real drop from a tracking artefact.
