Skip to content

fix(worker): debounce save queue so burst zones batch correctly - #1

Merged
guisea merged 2 commits into
masterfrom
fix/save-queue-debounce
Jun 1, 2026
Merged

guisea merged 2 commits into
masterfrom
fix/save-queue-debounce

Conversation

@guisea

@guisea guisea commented May 26, 2026 •

Copy link
Copy Markdown
Contributor

Problem

When DirectAdmin pushes multiple zones within the same second, the save worker was processing them in separate 1/1 batches instead of one grouped batch.

Root cause: The batch drain loop (get_nowait()) only worked if backend processing was slow enough to act as an accidental debounce gate. With fast backends, item 1 would finish and hit Empty before items 2 and 3 from the same burst had landed in the queue — each item then opened its own batch.

Fix

Adds BATCH_DEBOUNCE_SECONDS = 0.5 to worker.py. After the blocking get() fires on the first item, the worker sleeps 500 ms so the rest of the burst can accumulate. The existing drain loop then processes all of them in a single batch.

Also fixes from app.utils import … → from directdnsonly.app.utils import … (short-form import was unresolvable without a runtime sys.path hack, which blocked writing worker tests).

Tests (TDD)

New tests/test_worker.py:

  • test_burst_items_land_in_single_batch — 3 items arriving within the debounce window → asserts exactly 1 batch logged as 3/3
  • test_items_outside_debounce_window_start_new_batch — items separated by more than the window → asserts 2 separate batches
  • test_preloaded_queue_still_batches — 5 pre-loaded items → still drains as one batch

Test plan

  • All 156 tests pass (pytest tests/ -v)
  • Observe logs on next DA zone push burst — should see 📦 Batch complete — N/N with N > 1 instead of repeated 1/1

guisea added 2 commits May 26, 2026 22:28
Without a debounce, the save worker picked up the first queued item and
immediately started processing it.  If backend writes completed before the
next items arrived (common when backends are fast), get_nowait() returned
Empty and the batch closed at 1/1.  Subsequent items each opened their own
single-item batch instead of being grouped with the burst.

Adds BATCH_DEBOUNCE_SECONDS (default 0.5 s): after the blocking get() picks
up the first item, the worker sleeps briefly so burst items from DirectAdmin
(multiple zones pushed within the same second) can accumulate in the queue.
The existing drain loop then processes them all in a single batch.

Also fixes short-form `from app.utils import …` to fully-qualified
`from directdnsonly.app.utils import …` so the module is importable without
a sys.path hack, enabling the new test_worker.py suite.
@guisea

guisea commented Jun 1, 2026

Copy link
Copy Markdown
Contributor Author

Additional fixes pushed to this branch

Two more issues found while tracing a Let's Encrypt DNS-01 challenge failure (both cluster nodes received the push, but LE still failed).

Reload logs promoted to INFO

NSD and BIND reload success was logged at DEBUG — invisible at the default INFO level. Changed to INFO so reload events appear in production without needing DADNS_LOG_LEVEL=debug.

Reload failure now triggers retry

reload_zone() return value was silently discarded in _process_single_backend and _delete_single_backend. A failed reload returned True to the caller, the batch reported success, and NSD kept serving stale zone data with no log evidence. Reload failure now returns False, emitting a warning and scheduling a retry via the existing backoff queue.

Test mock corrected

reload_zone.return_value was None (falsy) in the mock backend fixture — this bug was invisible because the return value was ignored. Fixed to True; the two debounce tests now correctly validate the full success path.

@guisea
guisea merged commit 16cca1f into master Jun 1, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant