Dev Logs / Incident Classifier

Incident Classifier

A small language model that files an airline incident under one tag, and a reviewer who can change that tag before it is stored.

The operational problem

Baggage notes, refund cases, cabin complaints, and safety reports land in the same free-text pile. The expensive mistake is sending a safety report to the service queue. This project fine-tunes Qwen2.5-1.5B-Instruct to answer with exactly one of SAFETY_RISK, BAGGAGE_CLAIM, FINANCIAL_DISPUTE, or SERVICE_FEEDBACK. The model does not draft a reply.

Technical highlights

  • LoRA on q_proj and v_proj, rank 8, alpha 16, trained with TRL SFTTrainer for 3 epochs
  • ChatML prompts. Greedy decode, at most 16 new tokens, then a tag parser that prefers a known label
  • Seed-42 corpus: 50 unique texts per class, 5 held out per class (180 train, 20 test)
  • Streamlit desk caches the base model plus the adapter. The auditor accepts or replaces the tag and can add a note
  • Disagreements append to train.jsonl with the human label as the target, and are marked exported so a second click does not duplicate them
  • evaluate.py prints base versus LoRA on the first five test rows. There is no pytest suite

Business impact

The sidebar tracks incidents audited, the human correction rate, and whether a prediction is still waiting for approval. Corrections are the next training set. Agreed tags are not exported. The training text is synthetic, built from templates rather than a live customer-care extract.

The full write-up covers the data files, the training settings, the audit table, and what a single Streamlit process does and does not cover at desk volume.