Support Ticket Triage

In the Real World · Brief · Deep learning · Core · 2 to 3 days

9,000 support tickets, eight queues, and an answer key that lets you score yourself against the best result anyone could get. Build the router before you read the walkthrough.

The situation

Kesterly Software runs a support desk. Every ticket is read by a person and sent to one of 8 queues before anyone can answer it. You have 9,000 tickets from 2025-08-01 to 2026-06-26, already routed, and the desk wants to know how much of that job can be handed to a model.

The data

What the room believes

1. Ticket classification is a solved problem and any reasonable model will do.
2. Accuracy on a random split tells you how it will behave next month.
3. Pretrained embeddings are the cheap way to buy language understanding.
4. Whatever error is left after that is a modelling problem.

Definition of done

1. A verdict on each of the four beliefs, with the evidence.
2. At least three models compared, including a baseline that is not a model at all.
3. An evaluation that says something about next month’s tickets, not just this month’s, and a defence of how you split the data to get it.
4. A statement of the best score achievable on this data, and how you worked it out.
5. A rule for which tickets get routed automatically and which go to a person, with the cost of being wrong priced in.
6. A recommendation the support manager could act on, including what you would not automate.

If you want to go further

• Split the data by phrasing rather than at random and compare the two answers.
• Check whether your model’s confidence means anything, by bucketing predictions and measuring accuracy in each bucket.
• Work out the highest accuracy anyone could achieve on this file, and compare your model to that rather than to 100 per cent.
• Price a misroute, then find the confidence threshold that minimises total cost.
• Look at the tickets your model gets wrong at high confidence. They are a different problem from the ones it gets wrong at low confidence.

In the Real World · Deep learning · Core · 2 to 3 days

Routing support tickets is the classification problem everyone thinks is solved. A word counter gets 89.69 per cent on a random split and looks finished. On next month’s wording it gets 74.62. And the best possible score on this data is 93.05, not 100, which turns out to be the useful number.

Section 1Problem Definition

No code yet. Ticket routing looks like a benchmark task and behaves like an operations problem, and the gap between those two framings is where the value is.

The problem in one sentence

Some tickets say which queue they belong to and some genuinely do not, so the useful question is not how often the model is right but whether it knows which kind of ticket it is looking at.

Business objective

Route automatically where it is safe to, hand the rest to a person, and put a number on where that line should sit.

Hypotheses

1. H1. Ticket classification is a solved problem and any reasonable model will do.
2. H2. Accuracy on a random split tells you how it will behave next month.
3. H3. Pretrained embeddings are the cheap way to buy language understanding.
4. H4. Whatever error is left after that is a modelling problem.

Section 2Data Collection

import numpy as np
import pandas as pd

t = pd.read_csv('data/kesterly-tickets.csv')
key = pd.read_csv('data/kesterly-answer-key.csv')
d = t.merge(key, on='ticket_id')
d['text'] = d['subject'].fillna('') + '. ' + d['body']

print('tickets  :', len(d))
print('queues   :', d['queue'].nunique())
print('intents  :', d['intent'].nunique())
print('window   :', d['created_at'].min()[:10], 'to', d['created_at'].max()[:10])
print('body words: mean %.1f  median %d  p95 %d'
      % (d['body'].str.split().str.len().mean(),
         d['body'].str.split().str.len().median(),
         d['body'].str.split().str.len().quantile(0.95)))

tickets  : 9000
queues   : 8
intents  : 31
window   : 2025-08-01 to 2026-06-26
body words: mean 28.1  median 27  p95 47

Short messages, a median of 27 words. That matters for what follows: there is not much text per ticket, so a model that needs a lot of context to be useful will not get it here.

Section 3Data Preprocessing

3aDuplicates and schema checks

print('duplicate ticket ids :', int(d['ticket_id'].duplicated().sum()))
print('missing bodies       :', int(d['body'].isna().sum()))
print('queues outside the taxonomy :',
      int((~d['queue'].isin(sorted(d['queue'].unique()))).sum()))
print('answer key rows match tickets :', len(d) == len(t))

duplicate ticket ids : 0
missing bodies       : 0
queues outside the taxonomy : 0
answer key rows match tickets : True

3bHandling categorical mess

Three categorical columns. The queue is the target. The channel is nearly useless and is left out on purpose, because a model that learns to route web form tickets differently from email has learned about the form and not about the problem. The subject is the interesting one, and section 5a explains why.

3cDealing with outliers

Nothing to trim. The longest tickets are the ones raising two issues, and those are not outliers, they are 27.8% of the inbox and one of the things the model has to handle. Truncating at 96 tokens covers well past the 95th percentile of 47 words.

3dHandling missing values

Nothing is null, but 40.2% of subject lines say nothing at all: Help, Urgent, Question, no subject. That is not missing data to be imputed, it is a realistic feature of every support inbox, and it is the reason the body has to carry the decision.

3eHandling skewed data

print(d['queue'].value_counts(normalize=True).mul(100).round(1))

queue
Login and access                 23.9
Billing and invoices             17.5
Data import                      13.0
Performance and outages          11.7
Product feedback                 11.2
Account admin and permissions     9.0
Integrations and API              7.0
Reporting and exports             6.7
Name: proportion, dtype: float64

The largest queue takes 23.9% of tickets, so always guessing it scores 23.87 per cent. That is the floor every number below has to clear, and it clears it easily, which is exactly why accuracy alone is not going to settle anything.

3fData types and normalisation

Subject and body are joined into one string. No stemming, no stopword removal and no lowercasing, because two of the three models are pretrained and expect text the way people wrote it. Stripping it back is a habit from the word counting era that actively costs accuracy with a transformer.

Section 4Exploratory Data Analysis

4aTarget variable analysis

Eight queues, no hierarchy, one label per ticket. The interesting structure is not in the target column, it is in how much of the target the text actually determines.

4bNumerical variables

There is one numeric column worth anything and it is derived: message length. Long tickets are more likely to raise two issues, which makes them harder rather than more informative.

4cCategorical variables

print(d['channel'].value_counts())
print()
print('distinct subject lines :', d['subject'].nunique())
print('tickets whose subject is generic :',
      round(100 * d['subject'].isin(['Help', 'Question', 'Urgent', 'Issue', 'Problem',
                                     'Support needed', 'Hello', 'Query', 'Please help',
                                     'Not working', 'Assistance', 'Quick question',
                                     'Issue with the platform', '(no subject)']).mean(), 1))

Email          4926
Web form       2709
In app chat    1365

distinct subject lines : 107
tickets whose subject is generic : 40.2

4dRelationships between variables

This is the section that decides the project. The answer key says, for every ticket, whether the text contains enough to identify the queue.

und = (d['is_ambiguous'] == 1) & (d['deciding_detail_present'] == 0)
print('tickets with a single queue intent      :',
      round(100 * (d['is_ambiguous'] == 0).mean(), 1))
print('two queue intent, deciding detail given :',
      round(100 * ((d['is_ambiguous'] == 1) &
                   (d['deciding_detail_present'] == 1)).mean(), 1))
print('two queue intent, detail absent         :', round(100 * und.mean(), 1))
print()
print('best possible accuracy on the whole file:',
      round(100 * d['posterior_max'].mean(), 2))
print('best possible on the undecidable slice  :',
      round(100 * d.loc[und, 'posterior_max'].mean(), 2))

tickets with a single queue intent      : 71.7
two queue intent, deciding detail given : 12.7
two queue intent, detail absent         : 15.6

best possible accuracy on the whole file: 93.14
best possible on the undecidable slice  : 56.0

15.6% of tickets carry an intent that belongs to two queues without the sentence that separates them. On those, the best any model can do is pick the more common of the two, which is right 56.0 per cent of the time. That single fact caps the whole problem at 93.05 per cent.

4eTesting our hypotheses

4fSubgroups

Every ticket is written in one of two vocabularies that mean the same thing and share almost no words. 34.8% of the inbox uses the second one. Training on the first and testing on the second is the honest version of the question every classifier is really being asked, which is how it will do on the phrasings that have not arrived yet.

Section 5Feature Engineering

5aThe leakage trap

This project leaked, in the first build, and the ceiling is what caught it.

5bNew features

5cEncoding

Bigrams for the word counter, so that at least some short phrases survive. A 384 dimension sentence vector for the frozen embedding. Wordpiece tokens truncated at 96 for the transformer, which covers the 95th percentile ticket with room to spare.

5dFeature selection

None. With eight classes and short text there is nothing to prune, and the interesting choice is between representations rather than within one.

Section 6Model Selection

Section 7Model Training

7aBaselines

Two splits, run side by side throughout. A random quarter held out, and a split by vocabulary where the model trains on one way of phrasing things and is tested on the other.

random split      train 6,750  test 2,250  ceiling 93.41
unseen wording    train 5,868  test 3,132  ceiling 93.05

7bComparing candidates

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, f1_score
from sklearn.model_selection import train_test_split
from sentence_transformers import SentenceTransformer

QS = sorted(d['queue'].unique())
y = d['queue'].map({q: i for i, q in enumerate(QS)}).values
idx = np.arange(len(d))

# Two splits. The second trains on one vocabulary and tests on the other, which
# is the closest this file gets to asking about next month's tickets.
tr_r, te_r = train_test_split(idx, test_size=0.25, stratify=d['queue'], random_state=1)
tr_u = idx[d['phrasing_set'].values == 'A']
te_u = idx[d['phrasing_set'].values == 'B']
SPLITS = {'random': (tr_r, te_r), 'unseen wording': (tr_u, te_u)}

E = SentenceTransformer('sentence-transformers/all-MiniLM-L6-v2', device='cpu').encode(
    d['text'].tolist(), batch_size=128, normalize_embeddings=True)

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

def finetune(tr, te, epochs=3, seed=0):
    torch.set_num_threads(8); torch.manual_seed(seed); np.random.seed(seed)
    tok = AutoTokenizer.from_pretrained('distilbert-base-uncased')
    enc = tok(d['text'].tolist(), truncation=True, max_length=96,
              padding='max_length', return_tensors='pt')
    m = AutoModelForSequenceClassification.from_pretrained(
        'distilbert-base-uncased', num_labels=len(QS))
    opt = torch.optim.AdamW(m.parameters(), lr=3e-5)
    ids, att, yt = enc['input_ids'], enc['attention_mask'], torch.tensor(y)
    m.train()
    for _ in range(epochs):
        for i in range(0, len(tr), 32):
            b = torch.tensor(np.random.permutation(tr)[i:i + 32])
            opt.zero_grad()
            m(input_ids=ids[b], attention_mask=att[b], labels=yt[b]).loss.backward()
            opt.step()
    m.eval(); out = []
    with torch.no_grad():
        for i in range(0, len(te), 64):
            b = te[i:i + 64]
            out.append(m(input_ids=ids[b], attention_mask=att[b]).logits.softmax(-1))
    return torch.cat(out).numpy()

PROB = {}
for split, (tr, te) in SPLITS.items():
    ceiling = d['posterior_max'].iloc[te].mean()
    v = TfidfVectorizer(ngram_range=(1, 2), min_df=2, sublinear_tf=True)
    A = v.fit_transform(d['text'].iloc[tr])
    PROB[(split, 'counts')] = LogisticRegression(max_iter=3000, C=4.0).fit(
        A, y[tr]).predict_proba(v.transform(d['text'].iloc[te]))
    PROB[(split, 'embeddings')] = LogisticRegression(max_iter=3000, C=8.0).fit(
        E[tr], y[tr]).predict_proba(E[te])
    PROB[(split, 'distilbert')] = finetune(tr, te)
    for name in ('counts', 'embeddings', 'distilbert'):
        acc = accuracy_score(y[te], PROB[(split, name)].argmax(1))
        print('%-16s %-12s acc %6.2f  ceiling %6.2f  headroom %5.1f'
              % (split, name, 100 * acc, 100 * ceiling, 100 * acc / ceiling))

random           Most common queue                acc  23.87  ceiling  93.41  headroom  25.6
random           Word counts plus logistic        acc  89.69  ceiling  93.41  headroom  96.0
random           Frozen embeddings plus logistic  acc  88.62  ceiling  93.41  headroom  94.9
random           DistilBERT, fine tuned           acc  92.89  ceiling  93.41  headroom  99.4
unseen wording   Most common queue                acc  23.18  ceiling  93.05  headroom  24.9
unseen wording   Word counts plus logistic        acc  74.62  ceiling  93.05  headroom  80.2
unseen wording   Frozen embeddings plus logistic  acc  76.02  ceiling  93.05  headroom  81.7
unseen wording   DistilBERT, fine tuned           acc  81.86  ceiling  93.05  headroom  88.0

On a random split the word counter is within 3.2 points of the transformer and looks like the sensible choice. Change the split so the test tickets are phrased in words the model has never seen, and it falls 15.07 points while the transformer falls 11.03.

The frozen embedding is the interesting failure. It is worse than word counting on the random split, at 88.62 against 89.69, because collapsing a ticket into a single vector loses the ordering that 27.8% of tickets depend on. It generalises slightly better to new wording and still finishes behind the fine tuned model on both. Pretrained is not the same as adapted.

7cHyperparameter tuning

There is one parameter worth tuning and it is not in the model. It is the confidence above which a ticket gets routed without a person looking at it. Before that can be set, the confidence has to mean something.

te = SPLITS['unseen wording'][1]
Pb = PROB[('unseen wording', 'distilbert')]
sub = d.iloc[te].copy()
sub['conf'] = Pb.max(1)
sub['ok'] = Pb.argmax(1) == y[te]

und = (sub['is_ambiguous'] == 1) & (sub['deciding_detail_present'] == 0)
print('mean confidence, decidable tickets   :', round(sub.loc[~und, 'conf'].mean(), 3))
print('mean confidence, undecidable tickets :', round(sub.loc[und, 'conf'].mean(), 3))
print('accuracy, decidable                  :', round(100 * sub.loc[~und, 'ok'].mean(), 1))
print('accuracy, undecidable                :', round(100 * sub.loc[und, 'ok'].mean(), 1))
print('best possible, undecidable           :',
      round(100 * sub.loc[und, 'posterior_max'].mean(), 1))

mean confidence, decidable tickets   : 0.877
mean confidence, undecidable tickets : 0.713
accuracy, decidable                  : 87.9
accuracy, undecidable                : 49.9
best possible, undecidable           : 56.0

The model is well calibrated. Above 0.99 confidence it is right 100.0% of the time on 23.8% of tickets, and none of those are undecidable. Below 0.70 it is right less than half the time and 44.0% of that band is the undecidable slice. The ceiling is not spread evenly across the inbox, it is concentrated exactly where the model already says it is unsure.

7dFinal evaluation

Now the threshold can be priced. A person routing a ticket costs 40 seconds at 21.00 pounds an hour, so 0.233 pounds. A misroute costs 6 minutes of somebody picking it up, reassigning it and the customer waiting longer, so 2.10 pounds. Nine routing decisions per mistake.

RATE, HUMAN_SECONDS, MISROUTE_MINUTES = 21.0, 40, 6
C_HUMAN = RATE / 3600 * HUMAN_SECONDS
C_MISROUTE = RATE / 60 * MISROUTE_MINUTES
conf, corr = sub['conf'].values, sub['ok'].values

def cost(threshold, c_misroute=C_MISROUTE):
    auto = conf >= threshold
    wrong = (1 - corr[auto].mean()) if auto.sum() else 0.0
    return (auto.sum() * wrong * c_misroute + (~auto).sum() * C_HUMAN) / len(conf)

grid = [(round(t, 2), cost(round(t, 2))) for t in np.arange(0.30, 1.02, 0.02)]
best_t, best_c = min(grid, key=lambda r: r[1])
print('route everything by hand  : %.4f per ticket' % C_HUMAN)
print('route everything by model : %.4f per ticket' % cost(0.0))
print('threshold at %.2f          : %.4f per ticket' % (best_t, best_c))
print('automated share at that threshold : %.1f percent'
      % (100 * (conf >= best_t).mean()))

Route nothing automatically and every ticket costs 0.233 pounds in someone’s time. Route everything and the misroutes cost 0.381 pounds, which is worse than doing it all by hand. The minimum sits at 0.9.

The fallback catches what it is supposed to. Of the undecidable tickets, 86.3% end up in the human queue, and they make up 31.7% of it. A person sees roughly 17 tickets a day instead of all 40.

What the answer depends on

Every number above rests on a misroute costing 6 minutes. That assumption deserves a test rather than a footnote.

# Everything above rests on a misroute costing six minutes. Price that assumption.
print('break even misroute cost: %.2f pounds, or %.1f minutes'
      % (C_HUMAN / (1 - corr.mean()), C_HUMAN / (1 - corr.mean()) / RATE * 60))
for mins in (1, 2, 3, 4, 6, 10, 20):
    cm = RATE / 60 * mins
    everything = (1 - corr.mean()) * cm
    gated = min(cost(round(t, 2), cm) for t in np.arange(0.30, 1.02, 0.02))
    print('misroute %2d min   all model %.4f   all human %.4f   gated %.4f'
          % (mins, everything, C_HUMAN, gated))

Full automation only beats doing it by hand if a misroute costs less than 3.7 minutes, which for a support ticket that lands in the wrong queue is optimistic. The threshold policy is the cheapest option at every price tested, and its advantage grows as misroutes get more expensive. That is the finding to take away, because it does not depend on getting the assumption right.

Where the model still struggles

The weakest queues are the ones that share intents with another queue. Product feedback and Reporting and exports both take custom report requests, and nothing in the text separates them unless the writer says whether they have already looked. That is the ceiling showing up queue by queue rather than a defect to be fixed.

Section 8Documentation and Handoff

What to do, and who owns it

What not to do

• Do not report accuracy without a ceiling. 81.86 sounds like there is work left. Against 93.05 it does not.
• Do not evaluate on a random split. It flattered the word counter by 15.07 points and would have chosen the wrong model.
• Do not automate everything because the model is good. It costs 63 per cent more than automating nothing.
• Do not reach for frozen embeddings as a shortcut. Here they were worse than word counting and cost more to serve.
• Do not strip the text back. Stemming and stopword removal help a word counter and hurt a pretrained model.
• Do not treat the ambiguous tickets as a modelling failure. They are a form design problem, and one extra question fixes more than a bigger model.

Reproducibility

What to take from this

• Work out the ceiling before you tune anything. It is the difference between a model that needs work and a model that is finished.
• A random split measures memorisation as much as understanding. Hold out phrasings and the ranking changes.
• Pretrained is not adapted. A frozen sentence vector lost to word counting here, because it discards order.
• A calibrated model is worth more than an accurate one. The value is in knowing which tickets to hand back.
• Full automation is a choice, not a destination. Here it costs more than doing nothing, and the sensitivity test says that holds across any plausible misroute cost.
• Some error belongs to the form, not the model. One extra question beats another sprint.

The desk asked for a classifier and the useful deliverable was a threshold. The model is the easy half: a fine tuned transformer gets within 11.19 points of everything the text allows. The half that pays is knowing that 15.6% of the inbox cannot be routed from the words alone, and building the path that sends exactly those tickets to a person.