You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Add daily mirror check that repairs the CDN tomls - #1647
Adds a daily Mirror check workflow. The Mirror workflow only runs when a PR is merged, so a failed or cancelled run leaves the CDN serving old tomls until the next merge. From 2026-11-16 to 2026-11-18 every Mirror run failed on a broken rclone config and nobody noticed.
What it does (.github/scripts/mirror-check.sh, run by .github/workflows/mirror-check.yml):
Validates the rclone config first: the erigon-snapshots remote must exist with type s3. That catches the exact failure we just had.
Picks branches automatically, so new release branches need no workflow change:
every branch that already has tomls in the bucket (published by Mirror), and
new branches matching main release/* whose last commit is at most 90 days old. These get published.
Old branches that were never published are skipped, and bucket folders without a branch are only reported.
For each branch, compares the head's *.toml files (including webseed/) byte by byte with the bucket (rclone check --download) and copies the ones that differ. It then re-fetches the head and checks again. That verifies the copy and also picks up a merge that landed in the meantime. Tomls that exist only in the bucket are reported, never deleted.
Writes a per-branch table to the run summary. It posts to NOTIFICATION_WEBHOOK only when it repaired or published something, or failed, so there's no daily noise.
rclone errors are printed with 32-hex ids (the R2 account id in endpoints) redacted, because the job log is public.
Triggers
Daily at 05:17 UTC.
Manual run, with options: repair on/off, branches (only these globs), and test_notification (just post a test message, e.g. after rotating the webhook).
PRs that change the check run it report-only, without notifications.
Configuration (optional repository variables): MIRROR_CHECK_BRANCHES (globs for new branches), MIRROR_CHECK_EXCLUDE, MIRROR_CHECK_NEW_BRANCH_DAYS.
rclone is pinned to v1.76.0 (the version Mirror currently gets) and its download is checked against the published SHA256.
Tested locally, with a local git repo and a stub rclone:
Branch selection: in-sync branch; stale and missing files repaired (including webseed/); new branch published; old unpublished branch skipped; branch without tomls skipped; orphan folder reported; the exclude and only filters work.
Report-only: reports differences without changing anything.
Merge landing during the copy: the new head is copied too, and the bucket ends up equal to the branch.
Failures: a failed copy, a config without type, a failed bucket listing and a failed compare each exit 1 with an alert. The account id is redacted in the errors.
Notify step: messages for the repaired, failed and test cases, and very long alert lists are cut without losing the time and run link.
Real report-only run on this PR (run 44314665066), after all Mirror runs had been fixed: the 7 published branches are in sync (main, performance, release/3.0, 3.1, 3.4, 3.6, 3.10). The never-published release/2.60, 3.0-renamed and 3.0.3 were skipped. Tomls left in the bucket by removed chains show up as notes (e.g. main: 14 files only in the bucket).
The first real run caught a bug the stub tests could not: I had swapped the meaning of +/- in rclone's combined report, so those leftover tomls failed every comparison. Fixed in 7f4b5f8; the stub now follows rclone's documented symbols.
The Mirror workflow only runs when a PR is merged; a failed or cancelled
run leaves the CDN with old tomls until the next merge (2026-11-16..18:
every run failed on a broken rclone config and nobody noticed).
mirror-check runs daily (and on demand), compares the *.toml files of the
head of every published branch byte by byte with the bucket, copies the
ones that differ, verifies the copy, and posts to NOTIFICATION_WEBHOOK
when it had to repair something or failed. New branches matching
main/release/* are published automatically; bucket folders without a
branch are only reported. PRs that change the check run it report-only.
"+" marks files missing in the bucket and "-" files only in the bucket;
they were swapped, so tomls left in the bucket by removed chains made
every comparison fail. Keep rclone's per-file log only when a comparison
fails.
Separate report-only runs from production repair runs
.github/workflows/mirror-check.yml:34
GitHub retains only one pending run per concurrency group, even with cancel-in-progress: false. If the daily repair is queued behind another check, a subsequent report-only PR run can replace and cancel it, leaving the CDN unrepaired until the next schedule. Put report-only and notification-test runs in separate groups from production repairs so they cannot displace the daily run.
Reserve time for notification before the job timeout
.github/workflows/mirror-check.yml:69
Reaching the 30-minute job timeout can stop the job before Notify runs, even with always(), so a stalled check may produce no webhook alert. Give checkout, installation, and checking shorter step-level timeouts that leave time for notification before the job deadline.
Review effort: Balanced
This branch has not been deployed
No deployments
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a daily Mirror check workflow. The Mirror workflow only runs when a PR is merged, so a failed or cancelled run leaves the CDN serving old tomls until the next merge. From 2026-11-16 to 2026-11-18 every Mirror run failed on a broken rclone config and nobody noticed.
What it does (
.github/scripts/mirror-check.sh, run by.github/workflows/mirror-check.yml):erigon-snapshotsremote must exist with types3. That catches the exact failure we just had.main release/*whose last commit is at most 90 days old. These get published.*.tomlfiles (includingwebseed/) byte by byte with the bucket (rclone check --download) and copies the ones that differ. It then re-fetches the head and checks again. That verifies the copy and also picks up a merge that landed in the meantime. Tomls that exist only in the bucket are reported, never deleted.NOTIFICATION_WEBHOOKonly when it repaired or published something, or failed, so there's no daily noise.Triggers
repairon/off,branches(only these globs), andtest_notification(just post a test message, e.g. after rotating the webhook).Configuration (optional repository variables):
MIRROR_CHECK_BRANCHES(globs for new branches),MIRROR_CHECK_EXCLUDE,MIRROR_CHECK_NEW_BRANCH_DAYS.rclone is pinned to v1.76.0 (the version Mirror currently gets) and its download is checked against the published SHA256.
Tested locally, with a local git repo and a stub
rclone:webseed/); new branch published; old unpublished branch skipped; branch without tomls skipped; orphan folder reported; the exclude and only filters work.type, a failed bucket listing and a failed compare each exit 1 with an alert. The account id is redacted in the errors.Real report-only run on this PR (run 44314665066), after all Mirror runs had been fixed: the 7 published branches are in sync (
main,performance,release/3.0,3.1,3.4,3.6,3.10). The never-publishedrelease/2.60,3.0-renamedand3.0.3were skipped. Tomls left in the bucket by removed chains show up as notes (e.g.main: 14 files only in the bucket).The first real run caught a bug the stub tests could not: I had swapped the meaning of
+/-in rclone's combined report, so those leftover tomls failed every comparison. Fixed in 7f4b5f8; the stub now follows rclone's documented symbols.