feat(ast): add source-backed documentation facts and owner identities #1656

Objective

Introduce the minimal AST-owned model that preserves source evidence for extraction and the existing convention decisions for metric consumers.

Umbrella: #1654 · Planning ID: D02
Completion prerequisites: D01: #1655

Shared architecture and constraints

Follow the umbrella's shared contracts. AST recognition serves two consumers: exact legacy convention/coverage streaming for root LOC, and richer source documentation for owner lookup and local text. Source attachment, profile eligibility and physical metric ownership are separate. Preserve the legacy candidate view before run selection, original emission timing, and metrics-only early exits.

Documentation fragments, logical documents and convention-qualified owner edges have separate identities. Source offsets refer to the actual parsed buffer and included ranges. The index is caller-owned, immutable and source-bound; lookup validates tree/index identity. Complete text means local decoded/cleaned content with forward source mapping, not inherited or generator-expanded documentation.

Implementation touchpoints

• big-code-analysis-ast/src/roles/ and roles/mod.rs: shared candidate membership and context/projection contract.

• big-code-analysis-ast/src/lang_helpers/doc_text.rs: byte marker classifiers, split from the binding walker.

• big-code-analysis-ast/src/documentation.rs (new, with focused submodules as needed)

• big-code-analysis-ast/src/lib.rs

• big-code-analysis-ast/src/lang_helpers.rs

Implementation steps

• Map each fact field to existing role/context evidence and identify genuinely new attachment/text data; keep the role DSL and metric policy out of the public fact model.

• Represent included source fragments by half-open byte ranges with optional nodes. A logical document has ordered fragments and composition/role; associations are separate and convention-qualified.

• Represent concrete node owners and location-backed slots using enclosing context, role, ordinal and span. Distinguish resolved owner sets from uncertain alternatives and lexical containing scope from semantic owner.

• Keep raw occurrence identity independent of content equality; shared docs are stored once. Define equality index-locally and stable source ordering separately.

• Define internal intercrate streaming events for exact legacy decisions and coverage. They may be Rust-pub plumbing where needed, but never become a counts_as_dloc field on stable public documentation types.

• Provide sink/request semantics for extra evidence beyond legacy needs and view-qualified selection. Cache identity is internal and must distinguish semantic questions.

• Model text availability and forward-map coordinate types without committing to a full renderer or symbol table. Public-facing growing types must permit additive extension when re-exported later.

Tests and edge cases

• A gated-out DocTarget candidate, a callable that is not a space opener, and an ordinary comment of a DocComment-listed kind remain distinguishable without duplicating membership lists.

• One opaque node contains several POD fragments; one doc has two owners; equal text at different offsets remains two occurrences.

• Python primary/additional roles differ from literal-concatenation fragments.

• Concrete owner, location-backed slot, virtual target, file root and unresolved alternatives preserve all evidence.

• Empty doc, missing doc, disabled value and unavailable decoding are distinguishable.

• Hand-pinned ranges and source slices; don't derive expected values from returned facts.

Use independently specified expected owners, fragment counts/order, byte ranges, text and availability. Pair them with separate legacy metric assertions where relevant. Do not sort/trim observed results, rely only on facade/delegate equality, or accept generated snapshots without an expected-value anchor. Use verbatim source for EOF/CRLF cases. Perturb the decision-bearing expression and confirm the intended assertion fails; use the maintained utils/perturb.py only on tracked, clean targets, recording or stashing real work first. Multi-target cargo runs require --no-fail-fast; include the mutated crate and dependent crates before claiming a unique guard. Keep irreproducible fixtures/manifests in tests, maintained utils or an issue attachment, following .claude/rules/temp-files.md.

Acceptance criteria

• D01 examples fit the model without importing metric types into AST.
• Owners are the qualifying subset of syntactic targets; no existence boolean is treated as a complete owner list.
• No unsafe code, fabricated tree-sitter nodes, persisted node IDs, self-referential Ast fields or pointer-based output ordering.
• The internal legacy event and stable-documentation candidate types have a clear visibility boundary.

Verification and completion evidence

• Follow current role-census/role-coverage/role-bypass requirements and applicable role-parity-corpus checks. Preserve independent semantic fixtures; refresh .role-coverage-observed.txt when any sampled corpus gitlink changes.

• Put revisions, exact commands, tool versions and durable input links beside parity/coverage claims; host-local evidence paths are not citations.

• Recheck current source and the prerequisite contracts/baseline applicable to this issue; record intentional changes to the premises.

• Run focused production-path tests and affected language siblings. Use package-isolated -p big-code-analysis-ast / -p big-code-analysis feature builds; workspace feature unification is not isolation.

• Run the repository's required checks for the touched surfaces. For implementation changes: make worktree-setup, make pre-commit read from its BCA_GATE: verdict, and relevant test-language membership comparison. Add scaling and chain-audit checks for traversal changes; run make fuzz-check and seed replay for fuzz changes.

• Record exact test/perturbation/probe evidence in a completion comment and update this body. Review unexpected metric drift before snapshot acceptance; follow baseline and integration-submodule discipline if behavior deliberately changes.

• Land public API documentation, re-export/stability review and changelog entries with the originating API change. Keep unrelated CLI/web/Python surfaces outside this issue unless explicitly listed.

Revalidation against main (2026-11-04)

Source baseline inspected: 07822c69211fe851c452ac6c0ebbd99cf92589a5. Relevant landed work: PR #1717, PR #1769, PR #1803. These are implementation updates to the existing acceptance criteria; this review does not mark extraction work complete.

• Role declarations already provide candidate membership and source-aware projections. Fact types should refer to occurrences and resolved semantic evidence, not introduce another Role enum or table mirroring roles/mod.rs.

• Keep legacy convention events separate from public source facts. The role table's DocTarget node, Member binding, FuncOpener and normalized documentation owner are different abstractions. Tuple slots and multi-binding owners still need source-backed descriptors.

• Cache APIs now include visited-node state and multiple MemoQuestion/fold lanes. Result objects must not retain build-only mutable caches. Use source order for externally visible iteration, not pointer-key ordering.