Skip to content

Add daily mirror check that repairs the CDN tomls - #1647

Open
lystopad wants to merge 3 commits into
mainfrom
feature/lystopad/mirror-daily-check
Open

lystopad wants to merge 3 commits into
mainfrom
feature/lystopad/mirror-daily-check

Conversation

@lystopad

@lystopad lystopad commented Nov 18, 2026 •

Copy link
Copy Markdown
Member

Adds a daily Mirror check workflow. The Mirror workflow only runs when a PR is merged, so a failed or cancelled run leaves the CDN serving old tomls until the next merge. From 2026-11-16 to 2026-11-18 every Mirror run failed on a broken rclone config and nobody noticed.

What it does (.github/scripts/mirror-check.sh, run by .github/workflows/mirror-check.yml):

  • Validates the rclone config first: the erigon-snapshots remote must exist with type s3. That catches the exact failure we just had.
  • Picks branches automatically, so new release branches need no workflow change:
    • every branch that already has tomls in the bucket (published by Mirror), and
    • new branches matching main release/* whose last commit is at most 90 days old. These get published.
    • Old branches that were never published are skipped, and bucket folders without a branch are only reported.
  • For each branch, compares the head's *.toml files (including webseed/) byte by byte with the bucket (rclone check --download) and copies the ones that differ. It then re-fetches the head and checks again. That verifies the copy and also picks up a merge that landed in the meantime. Tomls that exist only in the bucket are reported, never deleted.
  • Writes a per-branch table to the run summary. It posts to NOTIFICATION_WEBHOOK only when it repaired or published something, or failed, so there's no daily noise.
  • rclone errors are printed with 32-hex ids (the R2 account id in endpoints) redacted, because the job log is public.

Triggers

  • Daily at 05:17 UTC.
  • Manual run, with options: repair on/off, branches (only these globs), and test_notification (just post a test message, e.g. after rotating the webhook).
  • PRs that change the check run it report-only, without notifications.

Configuration (optional repository variables): MIRROR_CHECK_BRANCHES (globs for new branches), MIRROR_CHECK_EXCLUDE, MIRROR_CHECK_NEW_BRANCH_DAYS.

rclone is pinned to v1.76.0 (the version Mirror currently gets) and its download is checked against the published SHA256.

Tested locally, with a local git repo and a stub rclone:

  • Branch selection: in-sync branch; stale and missing files repaired (including webseed/); new branch published; old unpublished branch skipped; branch without tomls skipped; orphan folder reported; the exclude and only filters work.
  • Report-only: reports differences without changing anything.
  • Merge landing during the copy: the new head is copied too, and the bucket ends up equal to the branch.
  • Failures: a failed copy, a config without type, a failed bucket listing and a failed compare each exit 1 with an alert. The account id is redacted in the errors.
  • Notify step: messages for the repaired, failed and test cases, and very long alert lists are cut without losing the time and run link.

Real report-only run on this PR (run 44314665066), after all Mirror runs had been fixed: the 7 published branches are in sync (main, performance, release/3.0, 3.1, 3.4, 3.6, 3.10). The never-published release/2.60, 3.0-renamed and 3.0.3 were skipped. Tomls left in the bucket by removed chains show up as notes (e.g. main: 14 files only in the bucket).
The first real run caught a bug the stub tests could not: I had swapped the meaning of +/- in rclone's combined report, so those leftover tomls failed every comparison. Fixed in 7f4b5f8; the stub now follows rclone's documented symbols.

The Mirror workflow only runs when a PR is merged; a failed or cancelled
run leaves the CDN with old tomls until the next merge (2026-11-16..18:
every run failed on a broken rclone config and nobody noticed).

mirror-check runs daily (and on demand), compares the *.toml files of the
head of every published branch byte by byte with the bucket, copies the
ones that differ, verifies the copy, and posts to NOTIFICATION_WEBHOOK
when it had to repair something or failed. New branches matching
main/release/* are published automatically; bucket folders without a
branch are only reported. PRs that change the check run it report-only.
Copilot AI balanced review requested due to automatic review settings November 18, 2026 13:27
"+" marks files missing in the bucket and "-" files only in the bucket;
they were swapped, so tomls left in the bucket by removed chains made
every comparison fail. Keep rclone's per-file log only when a comparison
fails.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The check can report success after comparison errors or branch changes, while concurrent mirror runs can leave the CDN stale.

8 open issues
What changed in this PR

Adds a recovery check alongside the merge-triggered Mirror workflow to detect and repair stale snapshot TOML manifests on the CDN.

Changes:

  • Adds daily and manual runs, plus report-only checks for same-repository PRs.
  • Discovers published and recent matching branches, compares TOML contents, and attempts repairs without deleting bucket-only files.
  • Pins and verifies the rclone download, generates branch summaries, and adds webhook notifications with a test mode.
File Description
.github/​workflows/​mirror-check.yml Configures triggers, rclone installation, execution options, and webhook notifications.
.github/​scripts/​mirror-check.sh Implements config validation, branch discovery, comparison, repair retries, reporting, and diagnostic redaction.

Review effort: Balanced

Comment thread .github/scripts/mirror-check.sh Outdated
compare() {
local combined=$WORK/combined
: >"$combined"
rclone check "$1" "$2" --include '*.toml' --download --combined "$combined" --log-level ERROR 2>>"$WORK/rclone.log" || true
(cd "$src" && find . -type f -name '*.toml' | sed 's|^\./||' | sort) >"$WORK/diff.txt"
fi
n_bad=$((n_missing + n_differ))
((n_bad == 0)) && break
- .github/scripts/mirror-check.sh

concurrency:
group: mirror-check
fail "$name" "cannot extract the branch"
return 0
fi
if [[ $(toml_count "$src") == 0 ]]; then
report "| \`$name\` | **OUT OF SYNC**, not repaired: $n_bad files ($(file_list "$WORK/diff.txt")) | |"
alert "$name: $n_bad toml files on the CDN differ from the branch head, not repaired: $(file_list "$WORK/diff.txt")"
return 0
fi
alert "$name: $n_bad toml files on the CDN differ from the branch head, not repaired: $(file_list "$WORK/diff.txt")"
return 0
fi
if ! rclone copy "$src" "$REMOTE/$name" --include '*.toml' --ignore-times --log-level ERROR 2>>"$WORK/rclone.log"; then
Comment on lines +167 to +170
if ! rclone listremotes --long | awk -v r="$remote_name:" -v t="$REMOTE_TYPE" '$1 == r && $2 == t { ok = 1 } END { exit !ok }'; then
echo "::error::rclone config has no remote '$remote_name' of type '$REMOTE_TYPE'"
alert "rclone config has no remote '$remote_name' of type '$REMOTE_TYPE' (check the MIRROR_CONF secret)"
exit 1
Comment on lines +105 to +121
if [ -z "$WEBHOOK_URL" ]; then
echo "::warning::NOTIFICATION_WEBHOOK is not set, notification not sent"
exit 0
fi
if [ "$TEST" = true ]; then
title="Mirror check: test notification"
elif [ "$JOB_STATUS" = failure ]; then
title="Mirror check FAILED"
else
title="Mirror check found differences on the CDN"
fi
details=""
[ -s "$ALERT" ] && details=$(head -n 15 "$ALERT" | cut -c1-300 | head -c 1500)
msg=$(printf '%s\n%s\nTime: %s\nRun: %s' "$title" "$details" "$(date -u '+%Y-%m-%d %H:%M:%S UTC')" "$RUN_URL" | sed '/^$/d')
jq -n --arg content "$msg" '{content: $content}' \
| curl -fsS --retry 3 --max-time 20 -H 'Content-Type: application/json' -d @- "$WEBHOOK_URL" >/dev/null \
|| echo "::warning::notification could not be sent"
Copilot AI balanced review requested due to automatic review settings November 18, 2026 13:38

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The check and existing Mirror workflow lack a shared per-branch lock, allowing concurrent writes to leave the CDN stale after a verified repair.

8 open issues
2 previously missed issues

In code that hasn't changed since last review

Medium severity Separate report-only runs from production repair runs

.github/​workflows/​mirror-check.yml:34

GitHub retains only one pending run per concurrency group, even with cancel-in-progress: false. If the daily repair is queued behind another check, a subsequent report-only PR run can replace and cancel it, leaving the CDN unrepaired until the next schedule. Put report-only and notification-test runs in separate groups from production repairs so they cannot displace the daily run.

Medium severity Reserve time for notification before the job timeout

.github/​workflows/​mirror-check.yml:69

Reaching the 30-minute job timeout can stop the job before Notify runs, even with always(), so a stalled check may produce no webhook alert. Give checkout, installation, and checking shorter step-level timeouts that leave time for notification before the job deadline.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants