[Jun516]

Summary

JsonChunker first tokenizes its flattened JSON representation, then wraps the token chunks with JSON-specific metadata. The inherited token metadata contains format: "text", and buildMetadata merges the inherited extras last, overwriting the intended format: "json". Valid JSON chunks therefore silently report the wrong format even though sourceType is JSON and structural fields are present.

This PR removes the inherited format and sourceType keys before merging token chunk metadata. It retains token positions and JSON structural metadata, while invalid JSON still falls back to token chunking with format: "text".

Changes

• Exclude the inner TokenChunker's format and sourceType from JsonChunker's extras.
• Assert JSON format/source type on parsed JSON and text/token metadata on the invalid-JSON fallback.
• Add a patch Changeset for @voltagent/rag.

Reproduction and validation

• Examined the current upstream main source at 72a46c76b4507e1f98b41d5050a92bd30587eb70.
• Isolated TypeScript harness recreated the reviewed JsonChunker, TokenChunker, and buildMetadata implementations, substituting only minimal tokenization/position helpers because the complete monorepo dependencies were unavailable.
• Before fix: valid JSON produces format: "text"; the targeted assertion fails.
• After fix: TypeScript compilation and targeted assertions pass for valid JSON (json/json) and invalid JSON fallback (text/token).
• Patch verified to apply cleanly to the reviewed source snapshot and passed git diff --check.

Not run: official monorepo Vitest, package TypeScript build, lint, full CI, and integration tests. These still need real execution in the upstream/fork CI environment. The isolated harness is not a substitute for official tests.

Scope and prior work

Independent of #1459, which changes detectFormat routing for top-level arrays; this PR fixes metadata produced after a valid JSON payload has already reached JsonChunker.

AI use

Prepared with AI-assisted source review, duplication search, isolated reproduction, and patch verification. All validation limits are documented above.

---

Summary by cubic

Fixes JsonChunker metadata so valid JSON chunks report format: "json" and sourceType: "json" instead of inheriting "text"/"token" from the inner token chunker.

• Strips the token chunk's format and sourceType before merging JSON metadata, while keeping token positions and structural fields.
• Adds assertions for the JSON metadata path (including token start) and the invalid-JSON fallback, plus a patch Changeset for @voltagent/rag.

^{Written for commit fb8daaa. Summary will update on new commits.}

Summary by CodeRabbit

• Bug Fixes
  • JSON chunks now retain the correct format and source type metadata: valid JSON chunks are identified as JSON, while invalid-JSON fallback chunks are identified as text with a token source. Token position metadata is also preserved.

[changeset-bot]

🦋 Changeset detected

Latest commit: fb8daaa

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@voltagent/rag	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

• Configuration used: defaults
• Review profile: CHILL
• Plan: Advanced
• Run ID: 4c2ba01c-0ce6-410a-9da8-7b3b8b50e2da

📥 Commits

Reviewing files that changed from the base of the PR and between 146be7c and fb8daaa.

📒 Files selected for processing: 1

• packages/rag/src/__tests__/json-chunker.spec.ts

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

📝 Walkthrough

JsonChunker now excludes format and sourceType from extra metadata when building chunk metadata. Tests check metadata for valid and invalid JSON. A patch changeset records the update.

Changes: JSON Chunk Metadata

Layer / File(s)	Summary
Preserve JSON chunk metadata packages/rag/src/chunkers/json-chunker.ts, packages/rag/src/__tests__/json-chunker.spec.ts, .changeset/calm-otters-json-format.md	JsonChunker excludes format and sourceType from chunkExtra. Tests assert the expected metadata for valid and invalid JSON. A patch changeset describes the update.

Priority: ⬇️ Low

Estimated code review effort: 2 (Simple) | ~7 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to fb8da

Valid JSON chunks retain JSON format and source type along with token metadata, and invalid JSON keeps its text fallback. No actionable merge-blocking risk is evident in this change.

Pre-merge checks: 5

Check name	Status	Explanation
Title check		The title clearly and concisely describes the primary change: preserving JSON format metadata in JsonChunker.
Description check		The description is detailed and covers the current behavior, new behavior, implementation, tests, changeset, scope, and validation limits. It does not reproduce the checklist headings exactly or link …
Docstring Coverage		No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2…
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Description check

Explanation

The description is detailed and covers the current behavior, new behavior, implementation, tests, changeset, scope, and validation limits. It does not reproduce the checklist headings exactly or link a related issue, but the required information is mostly present.

Full details: Docstring Coverage

Explanation

No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 2 files.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[cubic-dev-ai]

All reported issues were addressed across 3 files

<sub>Reply with feedback, questions, or to request a fix.

View guided diff | Re-trigger cubic</sub>