Skip to content

Make the query list scale with query history; fix N+1 on the log page - #722

Merged
chrisclark merged 3 commits into
masterfrom
faster-query-list
Oct 18, 2026
Merged

chrisclark merged 3 commits into
masterfrom
faster-query-list

Conversation

@chrisclark

@chrisclark chrisclark commented Oct 17, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

The home page (query list) loaded the entire query log history on every view:

  • get_queryset() did prefetch_related("querylog_set"), fetching every log row for every query just to display a run count and the last run.
  • recently_viewed() looped over all of the user's logs in Python. Starting that loop evaluates the whole queryset.
  • str(q.database_connection) ran one extra query per row.

So the page gets slower as history grows. On a local benchmark (125 queries, 300k logs) it took 9.7 s. demo.sqlexplorer.io, with 1.15M logs, takes ~5.5 s to first byte, while its other pages take ~0.3 s.

Changes

  • Run count and last run (time and status) are computed in SQL, using Count plus Subquery annotations. A query that has never run still shows its creation time as a successful run, matching Query.last_run_log().
  • Recently viewed is now one aggregate (latest run per query, top N) plus one fetch.
  • select_related("created_by_user", "database_connection") replaces the prefetch and removes the per-row query.
  • Explicit order_by("title"), because a model's Meta.ordering isn't applied once the query has an aggregate.

Also: N+1 on the query log page

Each row on /logs/ looked up its run_by_user and database_connection separately, which meant 25–35 SQL queries per page. select_related brings that to a constant 5. With this PR, the query list, logs, favorites, connections, query detail and playground pages each run a fixed number of queries as data grows.

Results (same 300k-log benchmark)

  • 9.7 s → 0.9 s.
  • A constant 7 SQL queries, regardless of how many queries or logs exist.
  • The rendered list is identical to before: names, order, last-run dates and status, run counts, and recently viewed. I checked this by diffing the rendered HTML.

Tests

  • test_last_run_status: a never-run query falls back to its creation time and success, and a failed last run shows as failed.
  • test_recently_viewed_is_latest_run_of_distinct_queries: one entry per query, using the latest run, and only the user's own runs.
  • test_query_count_does_not_grow_with_queries_or_logs: this fails on master (9 → 14 SQL queries as rows are added) and passes here.
  • test_log_page_query_count_does_not_grow_with_rows: fails on master (17 vs 7) and passes here.
  • Full suite: 291 tests pass under settings and settings_base. The list view and log view tests also pass on Postgres 16. ruff==0.16.8 is clean.

🤖 Generated with Claude Code

chrisclark and others added 3 commits October 17, 2026 15:15
The query list prefetched every QueryLog for every query just to show a run
count and the last run, and recently_viewed() iterated all of the user's logs
in Python. Both load the full log history on each page view: with 300k logs
the page took ~9.7s; demo.sqlexplorer.io (1.15M logs) takes ~5.5s.

- Run count, last run time and last run status are annotated in SQL.
- Recently viewed queries are computed with one aggregate plus one fetch.
- select_related database_connection removes a per-row query.

Same 300k-log benchmark: 0.9s, and a constant 7 SQL queries regardless of
how many queries or logs exist. The rendered list is identical to before.

Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Each log row looked up its run_by_user and database_connection separately
(25-35 SQL queries per page); select_related brings it to a constant 5.

Co-Authored-By: Claude Code <noreply@anthropic.com>
@chrisclark chrisclark changed the title Make the query list page scale with query history Make the query list scale with query history; fix N+1 on the log page Oct 17, 2026
@chrisclark
chrisclark merged commit 533f892 into master Oct 18, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant