Skip to content

[release/3.6] mirror: notify webhook when mirroring fails - #1644

Merged
lystopad merged 1 commit into
release/3.6from
feature/lystopad/mirror-failure-notification-release-3.6
Nov 18, 2026
Merged

lystopad merged 1 commit into
release/3.6from
feature/lystopad/mirror-failure-notification-release-3.6

Conversation

@lystopad

Copy link
Copy Markdown
Member

Adds a "Notify about failure" step to the Mirror workflow. When any step of the mirror job fails, it posts the base branch, the merged PR, the UTC time and the run link to the NOTIFICATION_WEBHOOK secret:

Mirror of release/3.6 to CDN FAILED (after merging PR #1634 https://github.com/erigontech/erigon-snapshot/pull/1634)
Time: 2026-11-18 13:10:00 UTC
Run: https://github.com/erigontech/erigon-snapshot/actions/runs/43995086668/attempts/2

Mirror runs failed silently from 2026-11-16 to 2026-11-18 (the rclone config in MIRROR_CONF lost its type line), so the CDN served old tomls without anyone noticing.

  • Values are passed through env and the JSON is built with jq, so branch or PR text cannot break the script or the payload.
  • A missing secret or an unreachable webhook (4 attempts, 20s timeout) only logs a warning; the webhook URL is never printed.
  • mirror.yml runs from the PR's base branch, so the same change goes to main, release/3.6, release/3.10 and release/3.4 in separate PRs.

Tested locally: the YAML parses; the step's script against a local HTTP listener sends the expected JSON, warns without failing when the secret is empty or the webhook returns 500, and passes shell/JSON special characters in the branch name through as plain text.

Post the branch, PR, UTC time and run link to the NOTIFICATION_WEBHOOK
secret when any step of the mirror job fails. A missing secret or an
unreachable webhook only logs a warning.

(cherry picked from commit 26688e8)
Copilot AI balanced review requested due to automatic review settings November 18, 2026 13:21

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the snapshot-mirroring failure notification change.

No open issues 🎉

What changed in this PR

Adds failure notifications to the release/3.6 snapshot mirroring workflow so CDN update failures no longer go unnoticed.

Changes:

  • Sends the base branch, merged PR, UTC timestamp, and workflow run link to NOTIFICATION_WEBHOOK after a job failure.
  • Passes values through environment variables and builds the JSON payload with jq.
  • Adds bounded webhook retries and logs warnings if the secret is missing or delivery fails.
File Description
.github/​workflows/​mirror.yml Adds a failure-triggered webhook notification before housekeeping.

Review effort: Balanced

@lystopad lystopad self-assigned this Nov 18, 2026
@lystopad
lystopad merged commit 0818431 into release/3.6 Nov 18, 2026
2 checks passed
@lystopad
lystopad deleted the feature/lystopad/mirror-failure-notification-release-3.6 branch November 18, 2026 13:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants