Repository navigation
test: validate forecasts against 100 public repositories - #316
Merged
Merged
Conversation
Contributor
Author
|
Reran the complete CI workflow after the first attempt's jobs were cancelled. Attempt 2 passed the Windows Node 24 check, both Linux Node 22 shards, and Linux Node 24 shard 1. Linux Node 24 shard 0 reached the all-dependency audit after its tests, browser test, quality and mutation ratchets passed, then failed on the high-severity picomatch advisory GHSA-fc5q-39xc-vjpf. The local audit reproduces the finding. picomatch 4.0.7 is already in the base lockfile, is the latest published version as of this check, and the reviewed advisory lists no patched version. Follow-up #317 records the dependency path and verification requirements. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #125
Closes #147
Closes #172
Closes #198
Closes #224
Closes #225
Closes #241
Closes #199
Closes #200
Closes #258
Closes #272
What changed
Normalize Windows paths in project discovery and TypeScript diagnostics. Close the browser test server’s active connections during cleanup and bound the browser test to 60 seconds.
Replace the development tooling paths that required vulnerable picomatch with the TypeScript dead-code check and tested project-glob adapter from fix: replace development tooling paths that require picomatch #318. The all-dependency npm audit reports zero vulnerabilities.
Pin 100 complete public checkouts across five ecosystems and record the bundled-data forecast and runtime for each.
npm run validatereplays and compares the JSON at a fixed date without rewriting the snapshots.Review every finding in 20 results against the pinned declaration and vendor evidence. Publish the per-finding labels, coverage gaps, and performance results in the linked accuracy report.
Report all 13 reviewed false positives to Ebb #626. Five flag a Node.js 24 action under Node.js 20 retirement. Eight use a deadline for a different release line or service.
Point npm coverage gaps to their declaration lines. The reduced CSS.escape case verifies that a coverage gap without affected events produces cloudy weather.
Quality bar
The harness uses the existing checkout cache and invokes the shipping CLI. It pins bundled Ebb 0.2.0 and preserves the forecast JSON, replacing only the checkout path with the repository slug. Benchmark timings live outside the comparison. The corpus replay stays outside required CI.
Checks CI cannot make
changes/for user-visible changesVerification
The real forecast browser privacy test passed locally with the cleanup fix. Windows path regressions are exercised in the CI matrix.
npm run checkpassed.npm run validate -- --recordfollowed bynpm run validatereproduced all 100 snapshots.node dist/validation/report.jsverified reviewed findings against their snapshots and reproduced the published counts.node dist/validation/dates.jsfound zero differences between 50 finding dates and the bundled rules. The manual review checks the vendor evidence independently.node dist/validation/perf.js --recordmeasured the ten largest complete checkouts. The slowest forecast took 0.993 seconds and the highest peak RSS was 188,563,456 bytes.Mizzen checked and built the documentation site with zero errors. The published forecast page links to the accuracy report.
The CSS.escape regression failed on the previous code and passed after the location change.
Notes for reviewers
Includes the prerequisite tooling correction proposed separately in fix: replace development tooling paths that require picomatch #318 so this branch can run the complete CI gate. Dynamic module entry points are declared in dead-code.json. The parser adapter is tested against pinned typescript-eslint 8.72.1.
The manual sample is purposive. The report gives sample counts and leaves population precision and recall unestimated.
The pinned snapshots retain the upstream findings while Ebb #626 tracks the data corrections.