Skip to content

test: validate forecasts against 100 public repositories - #316

Merged
arminazimi merged 5 commits into
mainfrom
test/125-forecast-validation
Nov 17, 2026
Merged

arminazimi merged 5 commits into
mainfrom
test/125-forecast-validation

Conversation

@arminazimi

@arminazimi arminazimi commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Closes #125
Closes #147
Closes #172
Closes #198
Closes #224
Closes #225
Closes #241
Closes #199
Closes #200
Closes #258
Closes #272

What changed

  • Normalize Windows paths in project discovery and TypeScript diagnostics. Close the browser test server’s active connections during cleanup and bound the browser test to 60 seconds.

  • Replace the development tooling paths that required vulnerable picomatch with the TypeScript dead-code check and tested project-glob adapter from fix: replace development tooling paths that require picomatch #318. The all-dependency npm audit reports zero vulnerabilities.

  • Pin 100 complete public checkouts across five ecosystems and record the bundled-data forecast and runtime for each. npm run validate replays and compares the JSON at a fixed date without rewriting the snapshots.

  • Review every finding in 20 results against the pinned declaration and vendor evidence. Publish the per-finding labels, coverage gaps, and performance results in the linked accuracy report.

  • Report all 13 reviewed false positives to Ebb #626. Five flag a Node.js 24 action under Node.js 20 retirement. Eight use a deadline for a different release line or service.

  • Point npm coverage gaps to their declaration lines. The reduced CSS.escape case verifies that a coverage gap without affected events produces cloudy weather.

Quality bar

The harness uses the existing checkout cache and invokes the shipping CLI. It pins bundled Ebb 0.2.0 and preserves the forecast JSON, replacing only the checkout path with the repository slug. Benchmark timings live outside the comparison. The corpus replay stays outside required CI.

Checks CI cannot make

  • Tests added or updated for changed behavior
  • Changelog fragment added in changes/ for user-visible changes

Verification

  • The real forecast browser privacy test passed locally with the cleanup fix. Windows path regressions are exercised in the CI matrix.

  • npm run check passed.

  • npm run validate -- --record followed by npm run validate reproduced all 100 snapshots.

  • node dist/validation/report.js verified reviewed findings against their snapshots and reproduced the published counts.

  • node dist/validation/dates.js found zero differences between 50 finding dates and the bundled rules. The manual review checks the vendor evidence independently.

  • node dist/validation/perf.js --record measured the ten largest complete checkouts. The slowest forecast took 0.993 seconds and the highest peak RSS was 188,563,456 bytes.

  • Mizzen checked and built the documentation site with zero errors. The published forecast page links to the accuracy report.

  • The CSS.escape regression failed on the previous code and passed after the location change.

Notes for reviewers

  • Includes the prerequisite tooling correction proposed separately in fix: replace development tooling paths that require picomatch #318 so this branch can run the complete CI gate. Dynamic module entry points are declared in dead-code.json. The parser adapter is tested against pinned typescript-eslint 8.72.1.

  • The manual sample is purposive. The report gives sample counts and leaves population precision and recall unestimated.

  • The pinned snapshots retain the upstream findings while Ebb #626 tracks the data corrections.

@arminazimi

Copy link
Copy Markdown
Contributor Author

Reran the complete CI workflow after the first attempt's jobs were cancelled.

Attempt 2 passed the Windows Node 24 check, both Linux Node 22 shards, and Linux Node 24 shard 1. Linux Node 24 shard 0 reached the all-dependency audit after its tests, browser test, quality and mutation ratchets passed, then failed on the high-severity picomatch advisory GHSA-fc5q-39xc-vjpf.

The local audit reproduces the finding. picomatch 4.0.7 is already in the base lockfile, is the latest published version as of this check, and the reviewed advisory lists no patched version. Follow-up #317 records the dependency path and verification requirements.

Failing job · Advisory

@arminazimi
arminazimi merged commit 0cb2353 into main Nov 17, 2026
26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment