Skip to content

fix(format): keep soft penalty zero at threshold - #237

Merged
helloml0326 merged 1 commit into
agentscope-ai:mainfrom
Elon-Jiang:fix/soft-repetition-threshold
Nov 15, 2026
Merged

helloml0326 merged 1 commit into
agentscope-ai:mainfrom
Elon-Jiang:fix/soft-repetition-threshold

Conversation

@Elon-Jiang

Copy link
Copy Markdown
Contributor

OpenJudge Version

0.2.5, main 03c444567d5bb10429b10f4d2ad9df493b0edcc9.

Description

With use_soft_penalty=True, min_scaling=0.5, max_penalty=-2, and the real simple tokenizer, public aevaluate() produces:

Response Repetition rate Before After
alpha beta gamma alpha beta 0.4 0 0
alpha beta alpha beta 0.5 -1 0
alpha beta alpha beta alpha 0.6 -0.4 -0.4

The exact threshold falls through both comparisons and retains the unnormalized repetition rate. Consequently, a less repetitive response receives a larger penalty than a more repetitive one. Include equality in the zero-penalty branch so soft scaling starts only above the threshold. Hard penalties and the default zero threshold retain their behavior.

Add 11 public scoring cases for unigram/bigram/trigram thresholds, a second threshold, below/above/default controls, metadata, and hard penalties. Document the existing min_scaling parameter. No dependencies, model calls, replaced tokenizer, or service fixtures.

Validation:

  • Original code: 4 failing threshold cases, 7 passing controls.
  • pytest tests/graders/format/ tests/graders/text/ -q: 232 passed (three first-import jieba warnings).
  • pre-commit run --all-files --verbose: all hooks passed, including configured pytest 1221 passed, 139 skipped, five existing NumPy zero-variance warnings. The configured test hook excludes multimodal/generator/data/examples; skipped provider tests were not executed.
  • Public reproduction above returns 0, 0, -0.4 after the fix; git diff --check passes.

No existing related issue was found after checking all open PR files, repetition/ngram/soft-penalty reports and the file history. AI assistance: OpenAI Codex for investigation, implementation and validation.

Checklist

  • Code has been formatted with pre-commit run --all-files.
  • All configured tests are passing; exclusions/skips are described above.
  • Docstrings follow project style.
  • Related documentation has been updated.
  • Code is ready for review.

Co-authored-by: OpenAI Codex <noreply@openai.com>
@helloml0326
helloml0326 merged commit 86deed7 into agentscope-ai:main Nov 15, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants