Repository navigation
fix(grader): match repeated tools one-to-one - #235
Merged
helloml0326 merged 1 commit intoNov 13, 2026
Merged
helloml0326 merged 1 commit into
helloml0326 merged 1 commit into
Conversation
Match same-name tool calls using a maximum-weight one-to-one assignment. Keep step, count and legacy mode behavior unchanged. Assisted-by: OpenAI Codex
helloml0326
approved these changes
Nov 13, 2026
Contributor
Author
|
Thanks for reviewing and merging this! I appreciate your time. |
5 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenJudge Version
0.2.5, based on
1925e6d9f0ee3467dec07e867b26cec62a708145.Description
Strict step-by-step scoring (
strict_mode=True, use_jaccard_similarity=False) reuses the first prediction with a matching tool name for every reference call. With two callssearch(A), search(B)on both sides, it returns 0.75 instead of 1.0. Conversely, predictionssearch(A), search(B)against referencessearch(A), search(A)receive 1.0 because the first prediction is counted twice.Group calls by name and maximize total parameter F1 with a one-to-one assignment within each group. This keeps matching independent of order and avoids the suboptimal assignments possible with a greedy algorithm. The existing tool-count requirement, step boundaries, loose mode and Jaccard mode are preserved. SciPy is declared as a direct dependency (
>=1.8.0) forlinear_sum_assignment; it was already installed transitively through the mandatory scikit-learn dependency.Twelve regression cases cover exact/reordered matches, repeated predictions, unequal counts, a non-greedy optimum, and separation across tool names and steps. Documentation describes the matching behavior.
Validation
test_installed_codex_accepts_command_arguments_without_model_call: this Windows environment findscodex.CMD, but the existing subprocess invocation raisesFileNotFoundError. The unchanged NumPy constant-correlation tests emit five warnings on both versions.pip checkpassed.Validation used Windows CPython 3.13.5 and the project's declared editable dev dependencies (including SciPy 1.18.1). The complete pre-commit wrapper, other platforms/Python versions, minimum dependency versions, live API quality/integration tests, multimodal and generator test suites were not run. No model calls were made. Two unrelated documentation/UI image assets were unavailable in the materialized source; all Python source and test data were present. The full test command used the exclusions configured by the repository's pre-commit hook.
AI assistance
Implemented and checked with OpenAI Codex, including AI-assisted code, tests, and independent review, under the account owner's authorization.
Checklist
pre-commit run --all-filescommandThe first two boxes are deliberately unchecked: the configured checks were run directly, and the existing Windows CLI test failure is disclosed above.