feat(ast): provide local documentation text and forward source mappings #1660

Objective

Implement reusable composition, cleanup and mapping machinery for source-local documentation text.

Umbrella: #1654 · Planning ID: D06
Completion prerequisites: D02: #1656

Shared architecture and constraints

Follow the umbrella's shared contracts. AST recognition serves two consumers: exact legacy convention/coverage streaming for root LOC, and richer source documentation for owner lookup and local text. Source attachment, profile eligibility and physical metric ownership are separate. Preserve the legacy candidate view before run selection, original emission timing, and metrics-only early exits.

Documentation fragments, logical documents and convention-qualified owner edges have separate identities. Source offsets refer to the actual parsed buffer and included ranges. The index is caller-owned, immutable and source-bound; lookup validates tree/index identity. Complete text means local decoded/cleaned content with forward source mapping, not inherited or generator-expanded documentation.

Implementation touchpoints

• big-code-analysis-ast/src/lang_helpers/doc_text.rs: current marker/structural-tag byte classifiers; not a full local-text decoder.

• big-code-analysis-ast/src/documentation/text.rs (proposed)

• big-code-analysis-ast/src/documentation source-range types

• language literal-decoder integration points

Implementation steps

• Define source segment/availability behavior for continued comments using a bounded source-aware projection or an explicitly tracked grammar dependency, preserving legacy metric accounting separately.

• Distinguish raw syntax, decoded static values and cleaned local content in the API contract. Full local content is unavailable if any required fragment is dynamic, undecodable or unrepresentable; an optional segmented result preserves known parts and opaque gaps.

• Support copied, decoded-token, expanded-token and synthetic output origins. Removed prefixes/continuations yield no output segment; this delivery provides forward text-to-source mapping only.

• Range queries spanning fragments return ordered source ranges and explicit synthetic portions, never a bounding range that includes unrelated code/comments. Define UTF-8 boundary and empty-range behavior.

• Provide whole-logical-document cleanup hooks. Literal decoders belong to adapters; do not build a universal evaluator. Preserve markup, escaped tags, examples and reference directives literally.

• Use run-based mapping segments and reusable rendered results; state allocation/cost behavior. Per-language indentation/tab rules and CRLF treatment must be specified rather than silently unified.

• Integrate an actual comment source path with D05; helper-level text tests alone cannot complete the joint pipeline milestone.

Tests and edge cases

• Tcl/iRules # comment plus backslash-newline and following text, LF versus CRLF, Bash's non-continuing comments, and synthetic/source/decoded newlines with exact origins.

• Hand-pinned source, raw fragments, complete local text and origin ranges; no expected ranges derived from extractor output.

• Python-like "@au" "dit" maps @Audit to two source spans, while a decoded \x40 maps @ to the whole escape token.

• Raw strings preserve literal escapes; unknown gaps never create a token by concatenating known pieces.

• Source LF, escaped LF and synthetic join LF have distinct origins; tab expansion maps back to its source token.

• Removed delimiters/backslash continuation, ragged stars, indentation, Markdown hard-break whitespace, fenced examples, XML and POD markup.

• Empty valid docs, invalid UTF-8, CRLF/EOF, unsupported surrogate decoding, and queries splitting a UTF-8 scalar.

Use independently specified expected owners, fragment counts/order, byte ranges, text and availability. Pair them with separate legacy metric assertions where relevant. Do not sort/trim observed results, rely only on facade/delegate equality, or accept generated snapshots without an expected-value anchor. Use verbatim source for EOF/CRLF cases. Perturb the decision-bearing expression and confirm the intended assertion fails; use the maintained utils/perturb.py only on tracked, clean targets, recording or stashing real work first. Multi-target cargo runs require --no-fail-fast; include the mutated crate and dependent crates before claiming a unique guard. Keep irreproducible fixtures/manifests in tests, maintained utils or an issue attachment, following .claude/rules/temp-files.md.

Acceptance criteria

• Complete means complete local content, not generator-rendered or inherited documentation.
• Raw access survives text unavailability; mappings account for every output segment.
• No metric code consumes cleaned text for row counts.
• The D05+D06 production smoke test exercises collection through final mapping.

Verification and completion evidence

• Follow current role-census/role-coverage/role-bypass requirements and applicable role-parity-corpus checks. Preserve independent semantic fixtures; refresh .role-coverage-observed.txt when any sampled corpus gitlink changes.

• Put revisions, exact commands, tool versions and durable input links beside parity/coverage claims; host-local evidence paths are not citations.

• Recheck current source and the prerequisite contracts/baseline applicable to this issue; record intentional changes to the premises.

• Run focused production-path tests and affected language siblings. Use package-isolated -p big-code-analysis-ast / -p big-code-analysis feature builds; workspace feature unification is not isolation.

• Run the repository's required checks for the touched surfaces. For implementation changes: make worktree-setup, make pre-commit read from its BCA_GATE: verdict, and relevant test-language membership comparison. Add scaling and chain-audit checks for traversal changes; run make fuzz-check and seed replay for fuzz changes.

• Record exact test/perturbation/probe evidence in a completion comment and update this body. Review unexpected metric drift before snapshot acceptance; follow baseline and integration-submodule discipline if behavior deliberately changes.

• Land public API documentation, re-export/stability review and changelog entries with the originating API change. Keep unrelated CLI/web/Python surfaces outside this issue unless explicitly listed.

Revalidation against main (2026-11-04)

Source baseline inspected: 07822c69211fe851c452ac6c0ebbd99cf92589a5. Relevant landed work: PR #1804, PR #1772. These are implementation updates to the existing acceptance criteria; this review does not mark extraction work complete.

• doc_text.rs is now the byte-level marker/virtual-tag classifier module, not a complete decoder/cleaner. Reuse its recognition functions without confusing their booleans with normalized documentation content.

• Bare continuation rows can be absent from AST nodes. PR fix: batch 2026-11-03-2 (8 issues) #1804 added physical gap-row accounting for Bash/Tcl/iRules; fix(tcl): a \-continued comment parses its next row as code #1802 still records continued-comment text parsed as commands and CRLF rows parsed as words. Cleaned text and mappings must not infer a true boundary solely from those parser nodes.

• Keep parser-included-range gaps distinct from lexical continuation gaps. Required mapping tests use verbatim input as well as normalized file input; current drift tooling has a verbatim view suitable for preservation evidence.