Skip to content

fix(agent): ignore tool argument key order - #239

Merged
helloml0326 merged 1 commit into
agentscope-ai:mainfrom
Elon-Jiang:fix/tool-jaccard-key-order
Nov 16, 2026
Merged

helloml0326 merged 1 commit into
agentscope-ai:mainfrom
Elon-Jiang:fix/tool-jaccard-key-order

Conversation

@Elon-Jiang

Copy link
Copy Markdown
Contributor

OpenJudge Version

0.2.5, main 5d737c9.

Description

The default strict Jaccard grader gives 0.0 for the same tool call when JSON argument keys are reordered: {"city":"杭州","units":"metric"} versus {"units":"metric","city":"杭州"}. It compares Python dictionary representations, which preserve insertion order.

Serialize arguments with sorted JSON keys on both sides. This also handles nested objects and objects inside arrays, while preserving array order, different values, tool names and existing set semantics. Update the matching documentation.

Validation:

  • Public aevaluate regressions on unchanged main: 6 failed, 29 passed.
  • Tool grader tests after fix: 91 passed, 11 skipped.
  • pre-commit run --all-files --verbose: all hooks pass, including configured pytest 1253 passed, 139 skipped (five existing NumPy zero-variance warnings).
  • git diff --check: passed.

Both JSON-text and dictionary argument inputs run through the real public grader. No model calls, services or replaced implementations are involved. The configured test hook excludes multimodal/generator/data/examples; skipped provider tests were not executed. Existing #235 changes one-to-one step matching, not this Jaccard path.

AI assistance: OpenAI Codex prepared the change and ran validation.

Checklist

  • Code has been formatted with pre-commit run --all-files.
  • All configured tests pass; exclusions/skips are described above.
  • Docstrings follow project style.
  • Related documentation has been updated.
  • Code is ready for review.

@helloml0326
helloml0326 merged commit 8c8d95f into agentscope-ai:main Nov 16, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants