design(ast): define documentation association and extraction contracts #1655

Objective

Freeze an implementable source-documentation contract before introducing result types or migrating DLOC helpers.

Umbrella: #1654 · Planning ID: D01
Completion prerequisites: R09: #1699 is satisfied (merged in PR #1769); the role series #1690/#1701 is closed. Retain the native edge as provenance. This contract work is ready to begin against the current role/context APIs.

This issue is a design prerequisite. Update both the relevant issue bodies and an explanatory comment when decisions change. Prefer recording task-specific decisions in the tracker; durable consumer contracts will land with the API and guide.

Shared architecture and constraints

Follow the umbrella's shared contracts. AST recognition serves two consumers: exact legacy convention/coverage streaming for root LOC, and richer source documentation for owner lookup and local text. Source attachment, profile eligibility and physical metric ownership are separate. Preserve the legacy candidate view before run selection, original emission timing, and metrics-only early exits.

Documentation fragments, logical documents and convention-qualified owner edges have separate identities. Source offsets refer to the actual parsed buffer and included ranges. The index is caller-owned, immutable and source-bound; lookup validates tree/index identity. Complete text means local decoded/cleaned content with forward source mapping, not inherited or generator-expanded documentation.

Implementation touchpoints

• big-code-analysis-ast/src/roles/<lang>.rs — the seven documentation rows per language (Comment, DocComment, Docstring, DocTarget, DocRunOpener, DocRunDirective, HoldsDocInBody), declared by R09: #1699; the C-family and JS-family fragments in roles/c_family.rs / roles/js_family.rs. An adapter's recognition set is these rows composed with the named context fns below — never a second kind inventory (roles/mod.rs, "The role set" and "Reading a row from a consumer").
• big-code-analysis-ast/src/checker.rs and checker/<lang>.rs — the context a row cannot see, as named fns beside the row read: tsdoc_target (a declaration under export / declare), Ruby's then-wrapper unwrapping over the ruby_yard_reads scope fold, Rust's tuple-struct field slot in is_doc_target, Doxygen's ERROR-holds-an-entry clause (lang_helpers::doxygen::doxygen_target), Lua's run-content decision in is_doc_run_opener, and each language's body-child test in holds_doc_comment_in_body.
• big-code-analysis-ast/src/lang_helpers/doc_comment.rs — the DocBinding (kept, skips, forward, gathers) that decides which doc comments of a stretch attach, and compatibility re-exports of the reusable marker tests now implemented in lang_helpers/doc_text.rs (java_doc, kotlin_doc, csharp_doc, phpdoc, javadoc_block, tsdoc_block, doxygen, pod_comment_region, go_directive, ruby_directive_opener, ruby_magic_comment, ldoc_opener).
• big-code-analysis-ast/src/lang_helpers/doxygen.rs — the entry projection (doxygen_entry), the DoxygenRow binding, the memoised conditional_declares, and doxygen_standalone / doxygen_member_after.
• big-code-analysis-ast/src/lang_helpers/ruby.rs — ruby_yard_reads (where YARD's handlers run) and ruby_documented_call (the attr* / alias_method / visibility-wrapped def forms), read by the Ruby rows and Checker alike.
• big-code-analysis-ast/src/lang_helpers/python.rs — python_is_documentation_string (the Docstring gate), python_documentation_string_run, and the Attributes / assignment-target helpers for the attribute-docstring form.
• STABILITY.md
• the umbrella and child issue bodies

Implementation steps

• Freeze the row/context/profile boundary and current baseline exceptions using the landed role-series reference before approving D02 interfaces.

• Specify recognition, profile-qualified selection, source attachment, publication eligibility and exact legacy reading as independent dimensions. The AST emits the old reading internally; root does not infer DLOC from public axes.

• Choose named default profiles and conflict behavior per language, especially PHP reflection/phpDocumentor and Ruby YARD/RDoc. Source-attached private/nested docs remain discoverable; publication is known eligibility under stated configuration or unknown. Generic proximity inference is opt-in; iRules gets a named BCA convention.

• Define fragments, documents, association edges, location-backed owners, roles, conditional candidates, forward-reference finalization, identity and source ordering. Define absent docs versus unsupported owner, foreign handle, disabled parser and unsupported convention.

• Specify exact raw/source-cleaned/decoded-local text semantics, complete versus segmented unavailable results, forward-only mappings, UTF-8 boundaries and per-language whole-document indentation rules. Local markup is preserved, not inherited or expanded.

• Specify immutable index context, trusted logical filename provenance, included-range barriers and parsed-buffer coordinates. Retain strict tree-qualified validation and separate index identity; no persistent Ast cache.

• Write an independent 25-LANG acceptance inventory with concrete promised forms. Every one of 23 source/dialect variants needs positive declaration/nonempty text coverage; each additional form gets its own exact result, including empty/disabled/unavailable. Helpers get explicit capability exceptions.

• Record source/compiler/tool references, versions/configuration and representative observations. Identify deliberate deviations and grammar/prelexing limits, including Java Unicode translation without ERROR nodes. Map each required case to its owning implementation issue.

Tests and edge cases

• Pin Comment-kind union versus gated DocComment; candidate DocTarget versus final Checker/owner; residual HoldsDocInBody versus full body decision; independently expected source and legacy results.

• Work through Java longer-old-doc plus empty-new-doc: source and legacy select different candidates before Last selection.

• Work through JS unattached in-body DLOC, Rust macro unknown-publication DLOC and Python additional documentation.

• Specify one source document for multiple qualifying fields, a conditional with alternative owners, a tuple-struct slot, Elixir multi-clause ownership and virtual named docs.

• Specify Python empty primary plus additional text, implicit literal concatenation, named escapes and lone surrogate unavailability; map a tag across disjoint literal ranges.

• Specify same-tree cross-index rejection, cloned/incremental tree rejection and shifted-node rejection; include restricted adopted-tree gaps.

• Specify conflicting conventions and present/absent logical filenames, including package-info.java and arbitrary display labels.

Use independently specified expected owners, fragment counts/order, byte ranges, text and availability. Pair them with separate legacy metric assertions where relevant. Do not sort/trim observed results, rely only on facade/delegate equality, or accept generated snapshots without an expected-value anchor. Use verbatim source for EOF/CRLF cases. Perturb the decision-bearing expression and confirm the intended assertion fails; use the maintained utils/perturb.py only on tracked, clean targets, recording or stashing real work first. Multi-target cargo runs require --no-fail-fast; include the mutated crate and dependent crates before claiming a unique guard. Keep irreproducible fixtures/manifests in tests, maintained utils or an issue attachment, following .claude/rules/temp-files.md.

Acceptance criteria

• Every choice above has a concrete contract and worked input/output example, not a deferred enum-name discussion.
• The source/form inventory is authored independently from runtime capabilities and linked to all child issues.
• Legacy streaming-only timing, sink capability/early-return mapping and cache question isolation are explicit.
• Public API scope remains source-local documentation; structural attachment tags are bounded, while general annotation parsing and cross-file expansion are outside this delivery.

Verification and completion evidence

• Follow current role-census/role-coverage/role-bypass requirements and applicable role-parity-corpus checks. Preserve independent semantic fixtures; refresh .role-coverage-observed.txt when any sampled corpus gitlink changes.

• Put revisions, exact commands, tool versions and durable input links beside parity/coverage claims; host-local evidence paths are not citations.

• Recheck current source and the prerequisite contracts/baseline applicable to this issue; record intentional changes to the premises.

• Run focused production-path tests and affected language siblings. Use package-isolated -p big-code-analysis-ast / -p big-code-analysis feature builds; workspace feature unification is not isolation.

• Run the repository's required checks for the touched surfaces. For implementation changes: make worktree-setup, make pre-commit read from its BCA_GATE: verdict, and relevant test-language membership comparison. Add scaling and chain-audit checks for traversal changes; run make fuzz-check and seed replay for fuzz changes.

• Record exact test/perturbation/probe evidence in a completion comment and update this body. Review unexpected metric drift before snapshot acceptance; follow baseline and integration-submodule discipline if behavior deliberately changes.

• Land public API documentation, re-export/stability review and changelog entries with the originating API change. Keep unrelated CLI/web/Python surfaces outside this issue unless explicitly listed.

Coordination with shared node-role recognition

Use the recognition definitions delivered by R09: #1699 under R00: #1690 when freezing this contract. The big-code-analysis-ast/src/roles/ declarations and the recognition helpers the adapters reuse are listed under Implementation touchpoints above (added when R09 landed). Do not create a second declaration-kind inventory in the extraction adapters.

Keep source-versus-legacy candidate views and selection separate where their semantics differ, as required by #1654. This dependency shares recognition; it does not make source documentation, lexical comments and legacy DLOC interchangeable.

Revalidation against main (2026-11-04)

Source baseline inspected: 07822c69211fe851c452ac6c0ebbd99cf92589a5. Relevant landed work: PR #1769, PR #1803, PR #1804. These are implementation updates to the existing acceptance criteria; this review does not mark extraction work complete.

• The native prerequisite refactor(checker): declare DocTarget, DocComment and Docstring rows ahead of the documentation epic #1699 is CLOSED and landed in PR refactor(roles,checker): declare the documentation rows and read them from the Checker #1769; refactor(ast): one role declaration per language for Checker, Getter and metric dispatchers #1690/docs(ast): document the role tables and close out #1701 closed through PR docs(ast): document the role tables and close out the series (#1701) #1803. Contract finalization is now unblocked. Preserve that edge as provenance instead of treating shared recognition as unfinished.

• Use roles/mod.rs as the contributor reference. Seven documentation rows now exist in every language. DocTarget is a candidate composed with contextual helpers; HoldsDocInBody is a residual, not the full body predicate. DocBinding/source-versus-legacy selection, normalized owners and text remain this epic's work.

• Freeze how the broader source interpretation reuses row kind evidence and named gates without changing existing legacy row verdicts. Absent Docstring/DocComment rows for new forms do not justify no-op adapters. Coordinate open refactor(roles): emit a per-role profile so a recognition row carries no paying predicates #1792/refactor(roles): add child_gated_by and CHAIN_GATED so byte-less forwards assert no gate reads the chain #1793/refactor(roles): decide a Parameter row for the TS/TSX after_code DocTarget subset #1796/refactor(roles): rename HoldsDocInBody or declare the full set and subtract in the Checker #1797 only where needed, rather than blocking on a second role-series redesign.

• refactor(ast): split doc_comment.rs into the binding walker and the text classifiers #1649 is complete: marker classifiers live in lang_helpers/doc_text.rs, re-exported through doc_comment.rs. The latter retains binding/run logic. Refresh terminology against current source, not the retired 'For the documentation epic' heading.

• Define all test and compatibility baselines against a new recorded main revision. fix(checker): exclude_tests attribute run stops at a comment #1647, fix(ast): exclude_tests reach keeps an item nested in an attribute value #1786, fix(metrics/loc): C family, Ruby, Elixir lose a bare \ row #1801 and fix(tcl): a \-continued comment parses its next row as code #1802 are outstanding semantic gaps; source documentation and legacy row expectations may intentionally differ there.