Skip to content

About

First-pass incident triage for application logs: error signatures, spike detection, recent changes. Pure Python, runs in the browser.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

1 Commit

Folders and files

Repository files navigation

Log Triage

tests

The first ten minutes of an incident investigation, automated.

Point it at an application log and it answers the questions I ask at the start of every investigation:

  • What's actually failing? Thousands of lines collapse into a handful of error signatures, with IDs, numbers, IPs and timestamps masked.
  • Since when, and is it real? Errors are bucketed over time and spikes are flagged against a robust baseline (median + MAD), so a long outage can't drag the baseline up and hide itself.
  • What changed? Deploys, rollbacks, restarts and config reloads just before the spike are surfaced, whatever their log level.
  • Where do I start? The earliest new warning or error before the spike.

Pure Python standard library, no dependencies. The same file runs in a terminal and in the browser.

▶ Try it live in your browser: zanchet.eu/projects/log-triage. Python runs locally via Pyodide, so your logs never leave the tab.

Example

$ python triage.py samples/checkout-outage.log

Parsed 2,060 of 2,060 lines covering 1h 59m (2026-03-14 09:00–11:00 UTC). 573 error-level
events collapse into 4 distinct error signatures. Incident 1: errors spiked 10:15–10:30 UTC
(peak 48/min vs a baseline of 0). Dominant error: payments-svc — "DB connection pool
exhausted (active=50/50, waiting=29)" (221×). It follows a change: payments-svc at 10:11:12 —
"Deploy started: version=2.14.0 (previous=2.13.4) by ci-pipeline". Earliest new signal: WARN
from payments-svc at 10:13:52 — "Connection pool nearing capacity (active=44/50) -
connections not returned after refund lookup". Start the investigation there.

Top 8 signatures
  count  level  service           pattern
    221  ERROR  payments-svc      DB connection pool exhausted (active=<n>/<n>, waiting=<n>)
    202  ERROR  checkout-svc      Upstream payments-svc returned <n> for order <n> after <n>
    133  ERROR  api-gateway       POST /v1/checkout <n> Gateway Timeout <n> client=<ip>
    ...

Usage

python triage.py app.log                   # human-readable report
python triage.py app.log --window 300      # 5-minute buckets
python triage.py app.log --json            # full report as JSON
kubectl logs deploy/api | python triage.py -

Requires Python 3.9+.

Supported formats

Plain text:

2026-03-14T10:15:02.120Z ERROR [payments-svc] DB connection pool exhausted
2026-03-14 10:15:02,120 [WARNING] checkout-svc: Slow query took 1200ms

JSON lines, with common field names (@timestamp/timestamp/time, level/severity/log.level, service/service.name/app, message/msg):

{"@timestamp": "2026-05-03T00:00:02Z", "level": "error", "service": "auth-svc", "message": "certificate has expired"}

Lines that can't be parsed are counted and skipped, not crashed on.

How it works

Step What happens
Parse Regex for text lines, json.loads for JSON lines; timestamps normalised to UTC
Normalise Variable parts masked (<uuid>, <ip>, <n>, <ts>, …) to form signatures
Detect Error counts per bucket; buckets above median + max(3 × 1.4826 × MAD, 3) are spikes; nearby spikes merge into one incident
Explain Per incident: top signatures in the window, changes in the 15 minutes before, earliest new signature, and a plain-English summary

Samples and tests

samples/ holds two synthetic logs with planted incidents, regenerated by make_samples.py:

  • checkout-outage.log: a payments deploy leaks DB connections and checkout fails
  • auth-cert-expiry.jsonl: an identity-provider certificate expires at midnight and logins fail
python -m unittest -v test_triage.py

The tests cover parsing, normalisation and spike detection, and check end to end that the triage points at the right cause in both samples.

License

MIT. Built by Lígia Zanchet.

About

First-pass incident triage for application logs: error signatures, spike detection, recent changes. Pure Python, runs in the browser.

Topics

Resources

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages