Skip to content

fix(text): compare numbers before float rounding - #238

Merged
helloml0326 merged 1 commit into
agentscope-ai:mainfrom
Elon-Jiang:fix/exact-number-accuracy
Nov 15, 2026
Merged

helloml0326 merged 1 commit into
agentscope-ai:mainfrom
Elon-Jiang:fix/exact-number-accuracy

Conversation

@Elon-Jiang

Copy link
Copy Markdown
Contributor

OpenJudge Version

0.2.5, main 03c444567d5bb10429b10f4d2ad9df493b0edcc9.

Description

NumberAccuracyGrader().aevaluate(response="9007199254740992", reference_response="9007199254740993") returns 1.0, although the values differ by 1 and the default absolute tolerance is 1e-6. Converting tokens to floats merges the two integers before comparison. Equivalent scientific values outside the float range (1e400 and 10e399) instead score 0.0 because both overflow to infinity.

Compare the original numeric tokens using standard-library Decimal arithmetic. Compute a nonnegative difference rounded upwards with a context that represents the tolerance exactly; this preserves inclusive tolerance boundaries even for differences beyond the default decimal precision, independently of the caller's decimal context. The exponent limits avoid expanding large powers into integers. Keep extraction syntax, ordered pairing, reference-count denominator and existing float-valued metadata unchanged. Preserve the previous NaN/infinite tolerance behavior.

Metadata remains a float representation for compatibility; it can still round or overflow. The score is calculated from the original tokens, as described in the updated documentation. No dependency or public API changes.

Validation:

  • Public reproduction: the large-integer mismatch now scores 0, equivalent scientific values score 1, and ordinary controls are unchanged.
  • Regression file: 10 failures, 17 passes before; 27 passes after, covering signed large integers, over/underflow, exact/near tolerance boundaries, decimal context isolation, partial scores and nonfinite tolerance controls.
  • 1008 public comparisons agree with independent standard-library Fraction arithmetic; a separate billion-exponent case completes without constructing a correspondingly large integer.
  • pytest tests/graders/text/ -q: 146 passed (three first-import jieba warnings).
  • pre-commit run --all-files --verbose: all hooks pass, including configured pytest 1232 passed, 139 skipped, five existing NumPy zero-variance warnings. The configured hook excludes multimodal/generator/data/examples; skipped provider tests were not executed.
  • git diff --check passes. No model/provider calls or service substitutes in the reproductions/regressions.

Existing #202 fixed notation parsing only. Complete relevant searches, all current open PR file lists, and full file history did not identify this comparison fix or an active claim. AI assistance: OpenAI Codex for investigation, implementation and validation.

Checklist

  • Code has been formatted with pre-commit run --all-files.
  • All configured tests are passing; exclusions/skips are described above.
  • Docstrings follow project style.
  • Related documentation has been updated.
  • Code is ready for review.

Co-authored-by: OpenAI Codex <noreply@openai.com>
@helloml0326
helloml0326 merged commit 5d737c9 into agentscope-ai:main Nov 15, 2026
1 check passed
@Elon-Jiang Elon-Jiang mentioned this pull request Nov 15, 2026
5 tasks done
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants