SPEC.md

Symphony Service Specification

Status: Draft v1 (language-agnostic)

Purpose: Define a service that orchestrates coding agents to get project work done.

Normative Language

The key words MUST, MUST NOT, REQUIRED, SHOULD, SHOULD NOT, RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in RFC 2119.

Implementation-defined means the behavior is part of the implementation contract, but this specification does not prescribe one universal policy. Implementations MUST document the selected behavior.

1. Problem Statement

Symphony is a long-running automation service that continuously reads work from a configured issue tracker, creates an isolated workspace for each issue, and runs a coding agent session for that issue inside the workspace.

The service solves four operational problems:

• It turns issue execution into a repeatable daemon workflow instead of manual scripts.
• It isolates agent execution in per-issue workspaces so agent commands run only inside per-issue workspace directories.
• It keeps the workflow policy in-repo (WORKFLOW.md) so teams version the agent prompt and runtime settings with their code.
• It provides enough observability to operate and debug multiple concurrent agent runs.

Implementations are expected to document their trust and safety posture explicitly. This specification does not require a single approval, sandbox, or operator-confirmation policy; some implementations target trusted environments with a high-trust configuration, while others require stricter approvals or sandboxing.

Important boundary:

• Symphony is a scheduler/runner and tracker reader.
• Ticket writes (state transitions, comments, PR links) are typically performed by the coding agent through provider-native tools executed by Symphony with the configured tracker credential.
• When tracker credentials are supplied through host-side secret references, the coding-agent child process does not need a duplicate tracker login or direct access to raw tracker credentials.
• A successful run can end at a workflow-defined handoff state (for example Human Review), not necessarily Done.

2. Goals and Non-Goals

2.1 Goals

• Poll the issue tracker on a fixed cadence and dispatch work with bounded concurrency.
• Maintain a single authoritative orchestrator state for dispatch, retries, and reconciliation.
• Create deterministic per-issue workspaces and preserve them across runs.
• Stop active runs when issue state changes make them ineligible.
• Recover from transient failures with exponential backoff.
• Load runtime behavior from a repository-owned WORKFLOW.md contract.
• Expose operator-visible observability (at minimum structured logs).
• Support tracker/filesystem-driven restart recovery without requiring a persistent database; exact in-memory scheduler state is not restored.

2.2 Non-Goals

• Rich web UI or multi-tenant control plane.
• Prescribing a specific dashboard or terminal UI implementation.
• General-purpose workflow engine or distributed job scheduler.
• Built-in business logic for how to edit tickets, PRs, or comments. (That logic lives in the workflow prompt and agent tooling.)
• Mandating strong sandbox controls beyond what the coding agent and host OS provide.
• Mandating a single default approval, sandbox, or operator-confirmation posture for all implementations.

3. System Overview

3.1 Main Components

1. Workflow Loader

   • Reads WORKFLOW.md.
   • Parses YAML front matter and prompt body.
   • Returns {config, prompt_template}.

2. Config Layer

   • Exposes typed getters for workflow config values.
   • Applies defaults and environment variable indirection.
   • Performs validation used by the orchestrator before dispatch.

3. Issue Tracker Adapter

   • Fetches candidate issues in active states.
   • Fetches current states for specific issue IDs (reconciliation).
   • Fetches terminal-state issues during startup cleanup.
   • Normalizes tracker payloads into a stable issue model.
   • MAY expose provider-native agent tools without adding provider-specific write APIs to the orchestrator.

4. Orchestrator

   • Owns the poll tick.
   • Owns the in-memory runtime state.
   • Decides which issues to dispatch, retry, stop, or release.
   • Tracks session metrics and retry queue state.

5. Workspace Manager

   • Maps issue identifiers to workspace paths.
   • Ensures per-issue workspace directories exist.
   • Runs workspace lifecycle hooks.
   • Cleans workspaces for terminal issues.

6. Agent Runner

   • Creates workspace.
   • Builds prompt from issue + workflow template.
   • Launches the coding agent app-server client.
   • Streams agent updates back to the orchestrator.

7. Status Surface (OPTIONAL)

   • Presents human-readable runtime status (for example terminal output, dashboard, or other operator-facing view).

8. Logging

   • Emits structured runtime logs to one or more configured sinks.

3.2 Abstraction Levels

Symphony is easiest to port when kept in these layers:

1. Policy Layer (repo-defined)

   • WORKFLOW.md prompt body.
   • Team-specific rules for ticket handling, validation, and handoff.

2. Configuration Layer (typed getters)

   • Parses front matter into typed runtime settings.
   • Handles defaults, environment tokens, and path normalization.

3. Coordination Layer (orchestrator)

   • Polling loop, issue eligibility, concurrency, retries, reconciliation.

4. Execution Layer (workspace + agent subprocess)

   • Filesystem lifecycle, workspace preparation, coding-agent protocol.

5. Integration Layer (selected tracker adapter)

   • API calls and normalization for tracker data.
   • Provider-native agent tools and centralized tracker authentication.

6. Observability Layer (logs + OPTIONAL status surface)

   • Operator visibility into orchestrator and agent behavior.

3.3 External Dependencies

• One configured issue tracker API.
• Local filesystem for workspaces and logs.
• OPTIONAL workspace population tooling (for example Git CLI, if used).
• Coding-agent executable that supports the targeted Codex app-server mode.
• Host environment authentication for the issue tracker and coding agent. Host-side tracker secret environment variables SHOULD NOT be inherited by the coding-agent child process.

4. Core Domain Model

4.1 Entities

4.1.1 Issue

Normalized schedulable work item used by orchestration, prompt rendering, and observability output. The name Issue is generic in this specification; an adapter MAY map it from a ticket, card, project item, or another provider-native work object.

Fields:

• id (string)
  • REQUIRED stable dispatch identity within the configured tracker scope.
  • Opaque to the orchestrator. It MAY be a project-item or board-entry ID instead of the provider's underlying ticket ID.
• native_ref (object or null)
  • OPTIONAL non-secret provider identifiers needed by provider-native tools.
  • Opaque to the orchestrator and preserved for prompt/tool context.
• identifier (string)
  • REQUIRED human-readable ticket key (example: ABC-123).
  • MUST be unique within the configured tracker scope because it names workspaces and operator-facing routes. An adapter spanning multiple namespaces MUST disambiguate it.
• title (string)
• description (string or null)
• priority (integer or null)
  • Lower numbers are higher priority in dispatch sorting.
• state (string)
  • REQUIRED current provider-native state name.
• branch_name (string or null)
  • Tracker-provided branch metadata if available.
• url (string or null)
• assignee_id (string or null)
• labels (list of strings)
  • Normalized to lowercase.
• blocked_by (list of blocker refs)
  • Best-effort provider metadata. Each blocker ref contains:
    • id (string or null)
    • identifier (string or null)
    • state (string or null)
• dispatchable (boolean)
  • REQUIRED adapter-derived eligibility for provider-specific rules that the generic scheduler cannot infer safely, such as assignment, board membership, or blocker semantics.
  • The orchestrator still applies configured state, label, claim, retry, and concurrency rules.
• created_at (timestamp or null)
• updated_at (timestamp or null)

4.1.2 Workflow Definition

Parsed WORKFLOW.md payload:

• config (map)
  • YAML front matter root object.
• prompt_template (string)
  • Markdown body after front matter, trimmed.

4.1.3 Service Config (Typed View)

Typed runtime values derived from WorkflowDefinition.config plus environment resolution.

Examples:

• poll interval
• workspace root
• active and terminal issue states
• concurrency limits
• coding-agent executable/args/timeouts
• workspace hooks

4.1.4 Workspace

Filesystem workspace assigned to one issue identifier.

Fields (logical):

• path (absolute workspace path)
• workspace_key (collision-resistant sanitized issue identifier)
• created_now (boolean, used to gate after_create hook)

4.1.5 Run Attempt

One worker execution for one issue. Crescendo distinguishes pre-model startup admission from model delivery attempts: startup failures do not consume delivery retry/effort or retirement budgets. Explicit session admission, rather than token usage, establishes the boundary. A separate persisted startup budget retries twice (10/20 seconds), then reports a startup block with one 30-minute recovery probe; relevant configuration changes permit immediate retry. Tracker items and dependencies are preserved, and capacity/claims are released during backoff. Startup history carries phase, hook/status, run/worker identity and bounded sanitized diagnostics.

Fields (logical):

• issue_id
• issue_identifier
• attempt (integer or null, null for first run, >=1 for retries/continuation)
• workspace_path
• started_at
• status
• error (OPTIONAL)

4.1.6 Live Session (Agent Session Metadata)

State tracked while a coding-agent subprocess is running.

Fields:

• session_id (string, <thread_id>-<turn_id>)
• thread_id (string)
• turn_id (string)
• codex_app_server_pid (string or null)
• last_codex_event (string/enum or null)
• last_codex_timestamp (timestamp or null)
• last_codex_message (summarized payload)
• codex_input_tokens (integer)
• codex_output_tokens (integer)
• codex_total_tokens (integer)
• last_reported_input_tokens (integer)
• last_reported_output_tokens (integer)
• last_reported_total_tokens (integer)
• turn_count (integer)
  • Number of coding-agent turns started within the current worker lifetime.

4.1.7 Retry Entry

Scheduled retry state for an issue.

Fields:

• issue_id
• identifier (best-effort human ID for status surfaces/logs)
• attempt (integer, 1-based for retry queue)
• due_at_ms (monotonic clock timestamp)
• timer_handle (runtime-specific timer reference)
• error (string or null)

4.1.8 Orchestrator Runtime State

Single authoritative in-memory state owned by the orchestrator.

Fields:

• poll_interval_ms (current effective poll interval)
• max_concurrent_agents (current effective global concurrency limit)
• running (map issue_id -> running entry)
• claimed (set of issue IDs reserved/running/retrying)
• retry_attempts (map issue_id -> RetryEntry)
• completed (set of issue IDs; bookkeeping only, not dispatch gating)
• codex_totals (aggregate tokens + runtime seconds)
• codex_rate_limits (latest rate-limit snapshot from agent events)

4.2 Stable Identifiers and Normalization Rules

• Issue ID
  • Use for tracker refresh calls and internal map keys.
  • Treat it as an opaque dispatch identity; do not assume it is the provider's underlying ticket ID.
• Native Ref
  • Preserve as opaque non-secret data for provider-native agent tools and prompt rendering.
  • Never use it as an orchestrator map key or interpret provider-specific fields in core logic.
• Issue Identifier
  • Use for human-readable logs and workspace naming.
  • Require uniqueness within the configured tracker scope.
• Workspace Key
  • Derive from issue.identifier by replacing any character not in [A-Za-z0-9._-] with _.
  • If sanitization changes the identifier, append a stable hash suffix of the original identifier with at least 64 bits of entropy using only allowed workspace-key characters, making keys for distinct identifiers that sanitize to the same text collision-resistant.
  • Use the resulting value for the workspace directory name.
• Normalized Issue State
  • Compare states after trimming surrounding whitespace and applying lowercase.
• Session ID
  • Compose from coding-agent thread_id and turn_id as <thread_id>-<turn_id>.

5. Workflow Specification (Repository Contract)

5.1 File Discovery and Path Resolution

Workflow file path precedence:

1. Explicit application/runtime setting (set by CLI startup path).
2. Default: WORKFLOW.md in the current process working directory.

Loader behavior:

• If the file cannot be read, return missing_workflow_file error.
• The workflow file is expected to be repository-owned and version-controlled.

5.2 File Format

WORKFLOW.md is a Markdown file with OPTIONAL YAML front matter.

Design note:

• WORKFLOW.md SHOULD be self-contained enough to describe and run different workflows (prompt, runtime settings, hooks, and tracker selection/config) without requiring out-of-band service-specific configuration.

Parsing rules:

• If file starts with ---, parse lines until the next --- as YAML front matter.
• Remaining lines become the prompt body.
• If front matter is absent, treat the entire file as prompt body and use an empty config map.
• YAML front matter MUST decode to a map/object; non-map YAML is an error.
• Prompt body is trimmed before use.

Returned workflow object:

• config: front matter root object (not nested under a config key).
• prompt_template: trimmed Markdown body.

5.3 Front Matter Schema

Top-level keys:

• tracker
• polling
• workspace
• hooks
• agent
• codex

Unknown keys SHOULD be ignored for forward compatibility.

Note:

• The workflow front matter is extensible. Extensions MAY define additional top-level keys without changing the core schema above.
• Extensions SHOULD document their field schema, defaults, validation rules, and whether changes apply dynamically or require restart.

5.3.1 tracker (object)

Fields:

• kind (string)
  • REQUIRED for dispatch.
  • Selects one implementation-supported tracker adapter.
• provider (object)
  • Default: {}.
  • Adapter-owned configuration such as endpoint, scope/project selector, and credentials.
  • Core Symphony MUST preserve unknown keys and MUST NOT prescribe one cross-provider credential or scope schema.
  • Each adapter MUST document its required keys, defaults, secret keys, $VAR_NAME support, and validation errors.
  • If a documented secret $VAR_NAME resolves to an empty string, treat that secret as missing.
• required_labels (list of strings)
  • Default: [].
  • An issue MUST contain every configured label to dispatch or continue.
  • Matching ignores case and surrounding whitespace.
  • A blank configured label matches no issue.
• excluded_labels (list of strings)
  • Default: [].
  • An issue carrying any configured label MUST NOT dispatch or continue. Adding one stops a running worker without workspace cleanup and releases a blocked claim.
  • Matching ignores case and surrounding whitespace.
• active_states (list of strings)
  • REQUIRED unless the selected adapter profile documents a default.
  • Values are provider-native state names compared case-insensitively by the scheduler.
• terminal_states (list of strings)
  • REQUIRED unless the selected adapter profile documents a default.
  • Values are provider-native state names compared case-insensitively by the scheduler.

5.3.2 polling (object)

Fields:

• interval_ms (integer)
  • Default: 30000
  • Changes SHOULD be re-applied at runtime and affect future tick scheduling without restart.

5.3.3 workspace (object)

Fields:

• root (path string or $VAR)
  • Default: <system-temp>/symphony_workspaces
  • ~ is expanded.
  • Relative paths are resolved relative to the directory containing WORKFLOW.md.
  • The effective workspace root is normalized to an absolute path before use.

5.3.4 hooks (object)

Fields:

• after_create (multiline shell script string, OPTIONAL)
  • Runs only when a workspace directory is newly created.
  • Failure aborts workspace creation.
• before_run (multiline shell script string, OPTIONAL)
  • Runs before each agent attempt after workspace preparation and before launching the coding agent.
  • Failure aborts the current attempt.
• after_run (multiline shell script string, OPTIONAL)
  • Runs after each agent attempt (success, failure, timeout, or cancellation) once the workspace exists.
  • Failure is logged but ignored.
• before_remove (multiline shell script string, OPTIONAL)
  • Runs before workspace deletion if the directory exists.
  • Failure is logged but ignored; cleanup still proceeds.
• timeout_ms (integer, OPTIONAL)
  • Default: 60000
  • Applies to all workspace hooks.
  • Invalid values fail configuration validation.
  • Changes SHOULD be re-applied at runtime for future hook executions.

5.3.5 agent (object)

Fields:

• max_concurrent_agents (integer)
  • Default: 10
  • Changes SHOULD be re-applied at runtime and affect subsequent dispatch decisions.
• max_turns (positive integer)
  • Default: 20
  • Limits the number of coding-agent turns within one worker session.
  • Invalid values fail configuration validation.
• max_retry_backoff_ms (integer)
  • Default: 300000 (5 minutes)
  • Changes SHOULD be re-applied at runtime and affect future retry scheduling.
• max_attempts (positive integer or null)
  • Default: null (retry failures indefinitely).
  • When a delivery failure-driven retry would exceed this attempt number, the issue is blocked instead of retried. The block is released when the issue's state or routability changes.
• max_concurrent_agents_by_state (map state_name -> positive integer)
  • Default: empty map.
  • State keys are normalized (trim + lowercase) for lookup.
  • Invalid entries (non-positive or non-numeric) are ignored.

5.3.6 codex (object)

Fields:

For Codex-owned config values such as approval_policy, thread_sandbox, and turn_sandbox_policy, supported values are defined by the targeted Codex app-server version. Implementors SHOULD treat them as pass-through Codex config values rather than relying on a hand-maintained enum in this spec. To inspect the installed Codex schema, run codex app-server generate-json-schema --out <dir> and inspect the relevant definitions referenced by v2/ThreadStartParams.json and v2/TurnStartParams.json. Implementations MAY validate these fields locally if they want stricter startup checks.

• command (string shell command)
  • Default: codex app-server
  • The runtime launches this command via bash -lc in the workspace directory.
  • The launched process MUST speak a compatible app-server protocol over stdio.
• approval_policy (Codex AskForApproval value)
  • Default: implementation-defined.
• thread_sandbox (Codex SandboxMode value)
  • Default: implementation-defined.
• turn_sandbox_policy (Codex SandboxPolicy value)
  • Default: implementation-defined.
• turn_timeout_ms (integer)
  • Default: 3600000 (1 hour)
• read_timeout_ms (integer)
  • Default: 5000
• stall_timeout_ms (integer)
  • Default: 300000 (5 minutes)
  • If <= 0, stall detection is disabled.

5.4 Prompt Template Contract

The Markdown body of WORKFLOW.md is the per-issue prompt template.

Rendering requirements:

• Use a strict template engine (Liquid-compatible semantics are sufficient).
• Unknown variables MUST fail rendering.
• Unknown filters MUST fail rendering.

Template input variables:

• issue (object)
  • Includes all normalized issue fields, including labels and blockers.
• attempt (integer or null)
  • null/absent on first attempt.
  • Integer on retry or continuation run.

Fallback prompt behavior:

• If the workflow prompt body is empty, the runtime MAY use a minimal default prompt (You are working on an issue from the configured tracker.).
• Workflow file read/parse failures are configuration/validation errors and SHOULD NOT silently fall back to a prompt.

5.5 Workflow Validation and Error Surface

Error classes:

• missing_workflow_file
• workflow_parse_error
• workflow_front_matter_not_a_map
• template_parse_error (during prompt rendering)
• template_render_error (unknown variable/filter, invalid interpolation)

Dispatch gating behavior:

• Workflow file read/YAML errors block new dispatches until fixed.
• Template errors fail only the affected run attempt.

6. Configuration Specification

6.1 Configuration Resolution Pipeline

Configuration is resolved in this order:

1. Select the workflow file path (explicit runtime setting, otherwise cwd default).
2. Parse YAML front matter into a raw config map.
3. Apply built-in defaults for missing OPTIONAL fields.
4. Resolve $VAR_NAME indirection for config values that explicitly contain $VAR_NAME, plus any adapter-owned fallback environment names documented for omitted provider fields.
5. Coerce and validate typed values.

Environment variables do not globally override YAML values. They are used only when a config value explicitly references them, or when an adapter profile documents a host-side fallback for an omitted provider field. Such a fallback is adapter-local, not a cross-provider convention.

Value coercion semantics:

• Path/command fields support:
  • ~ home expansion
  • $VAR expansion for env-backed path values
  • Apply expansion only to values intended to be local filesystem paths; do not rewrite URIs or arbitrary shell command strings.
• Relative workspace.root values resolve relative to the directory containing the selected WORKFLOW.md.

6.2 Dynamic Reload Semantics

Dynamic reload is REQUIRED:

• The software MUST detect WORKFLOW.md changes.
• On change, it MUST re-read and re-apply workflow config and prompt template without restart.
• The software MUST attempt to adjust live behavior to the new config (for example polling cadence, concurrency limits, active/terminal states, codex settings, workspace paths/hooks, and prompt content for future runs).
• Reloaded config applies to future dispatch, retry scheduling, reconciliation decisions, hook execution, and agent launches.
• Implementations are not REQUIRED to restart in-flight agent sessions automatically when config changes.
• Extensions that manage their own listeners/resources (for example an HTTP server port change) MAY require restart unless the implementation explicitly supports live rebind.
• Implementations SHOULD also re-validate/reload defensively during runtime operations (for example before dispatch) in case filesystem watch events are missed.
• Invalid reloads MUST NOT crash the service; keep operating with the last known good effective configuration and emit an operator-visible error.

6.3 Dispatch Preflight Validation

This validation is a scheduler preflight run before attempting to dispatch new work. It validates the workflow/config needed to poll and launch workers, not a full audit of all possible workflow behavior.

Startup validation:

• Validate configuration before starting the scheduling loop.
• If startup validation fails, fail startup and emit an operator-visible error.

Per-tick dispatch validation:

• Re-validate before each dispatch cycle.
• If validation fails, skip dispatch for that tick, keep reconciliation active, and emit an operator-visible error.

Validation checks:

• Workflow file can be loaded and parsed.
• tracker.kind is present and supported.
• The selected adapter accepts tracker.provider after documented defaults and $VAR resolution.
• codex.command is present and non-empty.

6.4 Core Config Fields Summary (Cheat Sheet)

This section is intentionally redundant so a coding agent can implement the config layer quickly. Extension fields are documented in the extension section that defines them. Core conformance does not require recognizing or validating extension fields unless that extension is implemented.

• tracker.kind: string, REQUIRED, selects one supported adapter
• tracker.provider: object, default {}, adapter-owned endpoint/scope/auth settings
• tracker.required_labels: list of strings, default []
• tracker.excluded_labels: list of strings, default []
• tracker.active_states: list of provider-native state names, adapter-defined default
• tracker.terminal_states: list of provider-native state names, adapter-defined default
• polling.interval_ms: integer, default 30000
• workspace.root: path resolved to absolute, default <system-temp>/symphony_workspaces
• hooks.after_create: shell script or null
• hooks.before_run: shell script or null
• hooks.after_run: shell script or null
• hooks.before_remove: shell script or null
• hooks.timeout_ms: integer, default 60000
• agent.max_concurrent_agents: integer, default 10
• agent.max_turns: integer, default 20
• agent.max_retry_backoff_ms: integer, default 300000 (5m)
• agent.max_attempts: positive integer or null, default null (unlimited)
• agent.max_concurrent_agents_by_state: map of positive integers, default {}
• codex.command: shell command string, default codex app-server
• codex.routing: optional explicit issue-label route map with label_prefix, default (model, effort), and labels mapping full label names to routes. A configured route overrides model at thread/start and effort at turn/start; no routing preserves Codex defaults. Unknown or conflicting route labels fail the run. The selected route is fixed for one agent run and may change on the next run. Optional ladder (routes ordered from cheapest to strongest) and escalation (non-decreasing, non-negative step offsets, one per item attempt; the last repeats) are configured together: the default and every label route must be a ladder step, and a run climbs escalation[item_attempt - 1] steps from its starting route, capped at the last step. The run keeps its starting route's label.
• The Codex process of every run is started with SYMPHONY_WORK_ITEM (the work item identifier, for example GH-12) and, when routing applies, SYMPHONY_SELECTED_MODEL_LABEL (the route label) in its environment, so commands the agent runs can attribute their work and check the selected route.
• codex.approval_policy: Codex AskForApproval value, default implementation-defined
• codex.thread_sandbox: Codex SandboxMode value, default implementation-defined
• codex.turn_sandbox_policy: Codex SandboxPolicy value, default implementation-defined
• codex.turn_timeout_ms: integer, default 3600000
• codex.read_timeout_ms: integer, default 5000
• codex.stall_timeout_ms: integer, default 300000

7. Orchestration State Machine

The orchestrator is the only component that mutates scheduling state. All worker outcomes are reported back to it and converted into explicit state transitions.

7.1 Issue Orchestration States

This is not the same as tracker states (Todo, In Progress, etc.). This is the service's internal claim state.

1. Unclaimed

   • Issue is not running and has no retry scheduled.

2. Claimed

   • Orchestrator has reserved the issue to prevent duplicate dispatch.
   • In practice, claimed issues are either Running or RetryQueued.

3. Running

   • Worker task exists and the issue is tracked in running map.

4. RetryQueued

   • Worker is not running, but a retry timer exists in retry_attempts.

5. Released

   • Claim removed because issue is terminal, non-active, missing, or retry path completed without re-dispatch.

Important nuance:

• A successful worker exit does not mean the issue is done forever.
• The worker MAY continue through multiple back-to-back coding-agent turns before it exits.
• After each normal turn completion, the worker re-checks the tracker issue state.
• If the issue is still in an active state, the worker SHOULD start another turn on the same live coding-agent thread in the same workspace, up to agent.max_turns.
• The first turn SHOULD use the full rendered task prompt.
• Continuation turns SHOULD send only continuation guidance to the existing thread, not resend the original task prompt that is already present in thread history.
• Once the worker exits normally, the orchestrator still schedules a short continuation retry (about 1 second) so it can re-check whether the issue remains active and needs another worker session.

7.2 Run Attempt Lifecycle

A run attempt transitions through these phases:

1. PreparingWorkspace
2. BuildingPrompt
3. LaunchingAgentProcess
4. InitializingSession
5. StreamingTurn
6. Finishing
7. Succeeded
8. Failed
9. TimedOut
10. Stalled
11. CanceledByReconciliation

Distinct terminal reasons are important because retry logic and logs differ.

7.3 Transition Triggers

• Poll Tick

  • Reconcile active runs.
  • Validate config.
  • Fetch candidate issues.
  • Dispatch until slots are exhausted.

• Worker Exit (normal)

  • Remove running entry.
  • Update aggregate runtime totals.
  • Schedule continuation retry (attempt 1) after the worker exhausts or finishes its in-process turn loop.

• Worker Exit (abnormal)

  • Remove running entry.
  • Update aggregate runtime totals.
  • Schedule exponential-backoff retry.

• Codex Update Event

  • Update live session fields, token counters, and rate limits.

• Retry Timer Fired

  • Re-fetch active candidates and attempt re-dispatch, or release claim if no longer eligible.

• Reconciliation State Refresh

  • Stop runs whose issue states are terminal or no longer active.

• Stall Timeout

  • Kill worker and schedule retry.

7.4 Idempotency and Recovery Rules

• The orchestrator serializes state mutations through one authority to avoid duplicate dispatch.
• claimed and running checks are REQUIRED before launching any worker.
• Reconciliation runs before dispatch on every tick.
• Restart recovery is tracker-driven and filesystem-driven (without a durable orchestrator DB).
• Startup terminal cleanup removes stale workspaces for issues already in terminal states.

8. Polling, Scheduling, and Reconciliation

8.1 Poll Loop

At startup, the service validates config, performs startup cleanup, schedules an immediate tick, and then repeats every polling.interval_ms.

The effective poll interval SHOULD be updated when workflow config changes are re-applied.

Tick sequence:

1. Reconcile running issues.
2. Run dispatch preflight validation.
3. Fetch candidate issues from tracker using active states.
4. Sort issues by dispatch priority.
5. Dispatch eligible issues while slots remain.
6. Notify observability/status consumers of state changes.

If per-tick validation fails, dispatch is skipped for that tick, but reconciliation still happens first.

8.2 Candidate Selection Rules

An issue is dispatch-eligible only if all are true:

• It has id, identifier, title, and state.
• Its state is in active_states and not in terminal_states.
• Its adapter-provided dispatchable value is true.
• It contains every label in tracker.required_labels.
• It contains no label in tracker.excluded_labels.
• It is not already in running.
• It is not already in claimed.
• Global concurrency slots are available.
• Per-state concurrency slots are available.

For refresh and continuation checks, issue_routable(issue) means only that adapter-provided dispatchable is true, all tracker.required_labels match, and no tracker.excluded_labels match. State, claims, and concurrency are checked separately by the surrounding algorithm.

Sorting order (stable intent):

1. priority ascending for values 1..4; all other integers and null sort after that bucket
2. created_at oldest first; null sorts last
3. identifier lexicographic tie-breaker

8.3 Concurrency Control

Global limit:

• available_slots = max(max_concurrent_agents - running_count, 0)

Per-state limit:

• max_concurrent_agents_by_state[state] if present (state key normalized)
• otherwise fallback to global limit

The runtime counts issues by their current tracked state in the running map.

8.4 Retry and Backoff

Retry entry creation:

• Cancel any existing retry timer for the same issue.
• Store attempt, identifier, error, due_at_ms, and new timer handle.

Backoff formula:

• Normal continuation retries after a clean worker exit use a short fixed delay of 1000 ms.
• Failure-driven retries use delay = min(10000 * 2^(attempt - 1), agent.max_retry_backoff_ms).
• Power is capped by the configured max retry backoff (default 300000 / 5m).
• Failure-driven retries (abnormal exit or stall) whose attempt number would exceed agent.max_attempts block the issue instead of scheduling a retry.

Retry handling behavior:

Due retries and retries held for admission participate in weighted slot demand; future backoff does not reserve a slot. A retry refreshes its slot policy before checking capacity.

1. Refresh the specific issue with fetch_issues_by_ids([issue_id]).
2. If not found, release claim.
3. If found in a terminal state, clean its workspace and release claim.
4. If found and still active and routable:
   • Dispatch if slots are available.
   • Otherwise requeue with error no available orchestrator slots.
5. If found but no longer active or routable, release claim without dispatch.

Note:

• Terminal-state workspace cleanup is handled by startup cleanup, active-run reconciliation, and retry refreshes that observe a terminal transition.
• ID refresh avoids treating a terminal, non-active, or newly unroutable issue as merely absent.

8.5 Active Run Reconciliation

Reconciliation runs every tick and has two parts.

Part A: Stall detection

• The orchestrator owns each worker phase; workers report entry into Codex and cleanup. Dispatch begins in workspace preparation, including after_create and before_run.
• Workspace preparation and after_run cleanup use the workspace operation/hook deadlines, not Codex inactivity. A hook timeout fails the attempt (cleanup failures remain best effort).
• During Codex startup and execution, compute elapsed_ms since:
  • last_codex_timestamp if any event has been seen, else
  • entry into the Codex phase, after workspace preparation.
• Startup also retains the finite codex.read_timeout_ms request/response deadline.
• Phase reports apply only to the current worker attempt. Reloads preserve phases; runtime restart cancels workers with their scheduler, and slots are released before redispatch.
• If elapsed_ms > codex.stall_timeout_ms, terminate the worker and queue a retry.
• If stall_timeout_ms <= 0, skip stall detection entirely.

Part B: Tracker state refresh

• Fetch current issue states for all running issue IDs.
• For each running issue:
  • If tracker state is terminal: terminate worker and clean workspace.
  • If tracker state is still active and routable: update the in-memory issue snapshot.
  • If tracker state is active but no longer routable: terminate worker without workspace cleanup.
  • If tracker state is neither active nor terminal: terminate worker without workspace cleanup.
• If state refresh fails, keep workers running and try again on the next tick.

8.6 Startup Terminal Workspace Cleanup

When the service starts:

1. Query tracker for issues in terminal states.
2. For each returned issue identifier, remove the corresponding workspace directory.
3. If the terminal-issues fetch fails, log a warning and continue startup.

This prevents stale terminal workspaces from accumulating after restarts.

9. Workspace Management and Safety

9.1 Workspace Layout

Workspace root:

• workspace.root (normalized absolute path)

Per-issue workspace path:

• <workspace.root>/<workspace_key>

Workspace persistence:

• Workspaces are reused across runs for the same issue.
• Successful runs do not auto-delete workspaces.

9.2 Workspace Creation and Reuse

Input: issue.identifier

Algorithm summary:

1. Derive workspace_key using Section 4.2, including the stable original-identifier hash when sanitization changes the identifier.
2. Compute workspace path under workspace root.
3. Ensure the workspace path exists as a directory.
4. Mark created_now=true only if the directory was created during this call; otherwise created_now=false.
5. If created_now=true, run after_create hook if configured.

Notes:

• This section does not assume any specific repository/VCS workflow.
• Workspace preparation beyond directory creation (for example dependency bootstrap, checkout/sync, code generation) is implementation-defined and is typically handled via hooks.

9.3 OPTIONAL Workspace Population (Implementation-Defined)

The spec does not require any built-in VCS or repository bootstrap behavior.

Implementations MAY populate or synchronize the workspace using implementation-defined logic and/or hooks (for example after_create and/or before_run).

Failure handling:

• Workspace population/synchronization failures return an error for the current attempt.
• If failure happens while creating a brand-new workspace, implementations MAY remove the partially prepared directory.
• Reused workspaces SHOULD NOT be destructively reset on population failure unless that policy is explicitly chosen and documented.

9.4 Workspace Hooks

Supported hooks:

• hooks.after_create
• hooks.before_run
• hooks.after_run
• hooks.before_remove

Execution contract:

• Execute in a local shell context appropriate to the host OS, with the workspace directory as cwd.
• On POSIX systems, sh -lc <script> (or a stricter equivalent such as bash -lc <script>) is a conforming default.
• Hook timeout uses hooks.timeout_ms; default: 60000 ms.
• Log hook start, failures, and timeouts.

Failure semantics:

• after_create failure or timeout is fatal to workspace creation.
• before_run failure or timeout is fatal to the current run attempt.
• after_run failure or timeout is logged and ignored.
• before_remove failure or timeout is logged and ignored.

9.5 Safety Invariants

This is the most important portability constraint.

Invariant 1: Run the coding agent only in the per-issue workspace path.

• Before launching the coding-agent subprocess, validate:
  • cwd == workspace_path

Invariant 2: Workspace path MUST stay inside workspace root.

• Normalize both paths to absolute.
• Require workspace_path to have workspace_root as a prefix directory.
• Reject any path outside the workspace root.

Invariant 3: Workspace key is sanitized.

• Only [A-Za-z0-9._-] allowed in workspace directory names.
• Replace all other characters with _.
• If replacement changes the identifier, append a stable original-identifier hash suffix with at least 64 bits of entropy so keys remain collision-resistant after sanitization.

10. Agent Runner Protocol (Coding Agent Integration)

This section defines Symphony's language-neutral responsibilities when integrating a Codex app-server. The Codex app-server protocol for the targeted Codex version is the source of truth for protocol schemas, message payloads, transport framing, and method names.

Protocol source of truth:

• Implementations MUST send messages that are valid for the targeted Codex app-server version.
• Implementations MUST consult the targeted Codex app-server documentation or generated schema instead of treating this specification as a protocol schema.
• If this specification appears to conflict with the targeted Codex app-server protocol, the Codex protocol controls protocol shape and transport behavior.
• Symphony-specific requirements in this section still control orchestration behavior, workspace selection, prompt construction, continuation handling, and observability extraction.

10.1 Launch Contract

Subprocess launch parameters:

• Command: codex.command
• Invocation: bash -lc <codex.command>
• Working directory: workspace path
• Transport/framing: the protocol transport required by the targeted Codex app-server version

Notes:

• The default command is codex app-server.
• Approval policy, sandbox policy, cwd, prompt input, and OPTIONAL tool declarations are supplied using fields supported by the targeted Codex app-server version.

RECOMMENDED additional process settings:

• Max line size: 16 MiB (Crescendo bounded frame assembly; RPC/control payloads remain capped at 4 MiB)

10.2 Session Startup Responsibilities

Reference: https://developers.openai.com/codex/app-server/

Startup MUST follow the targeted Codex app-server contract. Symphony additionally requires the client to:

• Start the app-server subprocess in the per-issue workspace.
• Initialize the app-server session using the targeted Codex app-server protocol.
• Create or resume a coding-agent thread according to the targeted protocol.
• Supply the absolute per-issue workspace path as the thread/turn working directory wherever the targeted protocol accepts cwd.
• Start the first turn with the rendered issue prompt.
• Start later in-worker continuation turns on the same live thread with continuation guidance rather than resending the original issue prompt.
• Supply the implementation's documented approval and sandbox policy using fields supported by the targeted protocol.
• Include issue-identifying metadata, such as <issue.identifier>: <issue.title>, when the targeted protocol supports turn or session titles.
• Advertise implemented client-side tools using the targeted protocol.

Session identifiers:

• Extract thread_id from the thread identity returned by the targeted Codex app-server protocol.
• Extract turn_id from each turn identity returned by the targeted Codex app-server protocol.
• Emit session_id = "<thread_id>-<turn_id>"
• Reuse the same thread_id for all continuation turns inside one worker run

10.3 Streaming Turn Processing

The client processes app-server updates according to the targeted Codex app-server protocol until the active turn terminates.

Completion conditions:

• Targeted-protocol turn completion signal -> success
• Targeted-protocol turn failure signal -> failure
• Targeted-protocol turn cancellation signal -> failure
• turn stream silence timeout (turn_timeout_ms) -> failure
• subprocess exit -> failure

Continuation processing:

• If the worker decides to continue after a successful turn, it SHOULD start another turn on the same live thread using the targeted protocol.
• The app-server subprocess SHOULD remain alive across those continuation turns and be stopped only when the worker run is ending.

Transport handling requirements:

• Follow the transport and framing rules of the targeted Codex app-server version.
• For stdio-based transports, keep protocol stream handling separate from diagnostic stderr handling unless the targeted protocol specifies otherwise.

10.4 Emitted Runtime Events (Upstream to Orchestrator)

The app-server client emits structured events to the orchestrator callback. Each event SHOULD include:

• event (enum/string)
• timestamp (UTC timestamp)
• codex_app_server_pid (if available)
• OPTIONAL usage map (token counts)
• payload fields as needed

Important emitted events include, for example:

• session_started
• startup_failed
• turn_completed
• turn_failed
• turn_cancelled
• turn_ended_with_error
• turn_input_required
• approval_auto_approved
• unsupported_tool_call
• notification
• other_message
• malformed

The orchestrator MUST tolerate non-object diagnostic payloads, including malformed protocol text. Model and turn bookkeeping MUST read only object envelopes and well-shaped fields; unrecognized content MUST NOT terminate the project or release its workers. Diagnostic content remains available in the last-message summary, and later valid notifications continue accounting and completion tracking. Optional protocol capture follows the same envelope validation.

10.5 Approval, Tool Calls, and User Input Policy

Approval, sandbox, and user-input behavior is implementation-defined.

Policy requirements:

• Each implementation MUST document its chosen approval, sandbox, and operator-confirmation posture.
• Approval requests and user-input-required events MUST NOT leave a run stalled indefinitely. An implementation MAY either satisfy them, surface them to an operator, auto-resolve them, or fail the run according to its documented policy.

Example high-trust behavior:

• Auto-approve command execution approvals for the session.
• Auto-approve file-change approvals for the session.
• Treat user-input-required turns as hard failure.

Unsupported dynamic tool calls:

• Supported dynamic tool calls that are explicitly implemented and advertised by the runtime SHOULD be handled according to their extension contract.
• If the agent requests a dynamic tool call that is not supported, return a tool failure response using the targeted protocol and continue the session.
• This prevents the session from stalling on unsupported tool execution paths.

Optional provider-native agent tool extension:

• An adapter MAY expose provider-native tools to the app-server session.
• The selected adapter's tool specs SHOULD be advertised during session startup using the protocol mechanism supported by the targeted Codex app-server version.
• Tool specs, adapter selection, and effective tracker settings MUST be bound to one session snapshot. A workflow reload applies to future sessions; it MUST NOT make an in-flight session advertise one provider and execute another.
• Tool names, schemas, and result payloads are adapter-owned. Symphony does not standardize a lowest-common-denominator CRUD API.
• The runtime MUST execute advertised tool calls host-side with the active adapter configuration and MUST NOT require the coding-agent child process to read raw tracker tokens from disk or environment.
• The runtime SHOULD pass the current normalized issue to the adapter as internal execution context. The adapter MAY use issue.id and issue.native_ref to preserve provider-specific richness without teaching the orchestrator provider semantics.
• Tracker credentials SHOULD NOT be inherited by the coding-agent child process. An adapter that resolves credentials from environment variables MUST declare authentication-related environment names for removal from local and remote child environments. Implementations SHOULD consult current provider and client documentation when identifying credential names and aliases, as these can change over time. Literal credentials in a repo-owned WORKFLOW.md remain readable to a child with workspace access and SHOULD NOT be used when this isolation matters.
• Unsupported tool names MUST return a structured failure result using the targeted protocol and continue the session.
• Each adapter that ships tools MUST document:
  • tool names and input schemas;
  • whether a tool can mutate tracker state;
  • scope/authorization behavior;
  • result and error semantics;
  • any provider-side idempotency or rate-limit expectations.

Minimal language-neutral adapter hooks for this OPTIONAL extension:

agent_tool_specs() -> list<ToolSpec>
secret_environment_names() -> list<string>
execute_agent_tool(name, arguments, context={issue}) -> ToolResult

ToolResult MUST distinguish success from failure and carry JSON-safe structured output that can be translated to the targeted app-server protocol. The context contains the normalized issue, never the credential.

User-input-required policy:

• Implementations MUST document how targeted-protocol user-input-required signals are handled.
• A run MUST NOT stall indefinitely waiting for user input.
• A conforming implementation MAY fail the run, surface the request to an operator, satisfy it through an approved operator channel, or auto-resolve it according to its documented policy.
• The example high-trust behavior above fails user-input-required turns immediately.

10.6 Timeouts and Error Mapping

Timeouts:

• codex.read_timeout_ms: request/response timeout during startup and sync requests
• codex.turn_timeout_ms: maximum silence interval while a turn stream is active; each app-server output resets it, so it is not a total turn runtime cap
• codex.stall_timeout_ms: enforced by orchestrator during Codex startup/execution based on event inactivity; workspace preparation and cleanup retain their own hook deadlines

Error mapping (RECOMMENDED normalized categories):

• codex_not_found
• invalid_workspace_cwd
• response_timeout
• turn_timeout
• port_exit
• response_error
• turn_failed
• turn_cancelled
• turn_input_required

10.7 Agent Runner Contract

The Agent Runner wraps workspace + prompt + app-server client.

Behavior:

1. Create/reuse workspace for issue.
2. Build prompt from workflow template.
3. Start app-server session.
4. Forward app-server events to orchestrator.
5. On any error, fail the worker attempt (the orchestrator will retry).

Note:

• Workspaces are intentionally preserved after successful runs.

11. Issue Tracker Integration Contract

The issue tracker boundary is deliberately small: a portable read kernel for scheduling plus OPTIONAL provider-native agent tools. Do not add generic comment/state/attachment CRUD merely to make providers look alike; those operations lose useful provider semantics and are not needed by the orchestrator.

11.1 REQUIRED Adapter Operations

An implementation MUST support these adapter operations:

1. fetch_issues_by_states(state_names)

   • Return normalized issues visible in the configured tracker scope and requested state names.
   • The adapter MUST apply provider-side scope selection and pagination.
   • Used with configured active states for candidate polling and terminal states for startup cleanup.
   • When used for candidate polling, include active scoped issues even when dispatchable=false; the scheduler owns that final filter.
   • The orchestrator applies required_labels, dispatchable, claims, retries, and concurrency after normalization.
   • An empty state_names list MUST return an empty result without a provider request.

2. fetch_issues_by_ids(issue_ids)

   • Return current normalized issue snapshots for the supplied opaque dispatch IDs.
   • Used for active-run reconciliation and stale-dispatch revalidation.
   • An empty issue_ids list MUST return an empty result without a provider request.
   • IDs no longer visible in the configured scope are omitted; the orchestrator treats omission as "no longer visible" rather than inventing a synthetic state.

Both operations return either ok(list<Issue>) or an adapter error. For portability, an adapter error SHOULD expose a stable category and human-readable message. An implementation MAY use a language-native tagged error, exception, tuple, or enum instead of a literal error object when its adapter profile documents how those public error forms map to category and message. The orchestrator relies only on success versus failure.

The operations are atomic from the scheduler's perspective after a paging or transport failure. For these rules, a record is malformed only when the adapter cannot produce the required normalized fields (id, identifier, title, state, and explicit dispatchable) or cannot produce a valid Issue after applying the optional-field fallback rules in Section 11.3. Unusable nullable or best-effort provider metadata MAY normalize to null, an empty list, or omitted best-effort entries; that fallback alone does not make a record malformed.

A state-list call MAY omit an individually malformed provider record because it was never safe to dispatch, and SHOULD log that omission. An ID-refresh call MUST fail instead of silently omitting a malformed requested record, because omission is meaningful. A successful fetch_issues_by_ids result is complete for that call. Output order is not significant, input IDs are treated as a set, and each dispatch ID appears at most once.

The refresh operation returns full normalized snapshots, not only state strings, because label, assignment, routing, and provider-specific dispatchability can change while a run is active.

11.2 Adapter Responsibilities

Each adapter owns:

• construction from the current effective tracker configuration, including active/terminal states;
• endpoint, authentication, transport, timeouts, pagination, and rate-limit handling;
• provider-specific scope selection (project, board, team, repository, query, or equivalent);
• mapping provider payloads into the normalized Issue fields in Section 4.1.1;
• choosing a stable dispatch identity and preserving any distinct underlying IDs in native_ref;
• deriving dispatchable from provider-specific routing rules;
• preserving provider-native state names while allowing case-insensitive scheduler comparison;
• OPTIONAL provider-native agent tools and their authorization boundary.

The orchestrator MUST NOT inspect provider payloads, assume that issue.id is an underlying ticket ID, or branch on provider-specific blocker, board, transition, or comment semantics.

Each adapter MUST publish a compact profile in implementation documentation, not only code, containing:

• exact supported tracker.kind value;
• exact tracker.provider keys, defaults, secret keys/environment names, and validation errors;
• scope selection, pagination behavior, and provider request limits;
• id and native_ref mapping;
• state, label, priority, timestamp, dispatchable, malformed-record, and optional-field normalization;
• provider-native tool names/schemas, mutation capability, scope, and result/error behavior if any;
• mapping from public language-native error forms to portable transport/auth/rate-limit error categories and human-readable messages.

11.3 Normalization Rules

Adapter output MUST satisfy Section 4.1.1. In addition:

• Every listed field MUST be present in the normalized record. Nullable fields use null; collection fields use an empty list when absent.
• id, identifier, title, and state MUST be non-empty strings.
• labels MUST be trimmed, lowercased strings; blank labels MUST be dropped and duplicate labels SHOULD be removed.
• priority MUST be an integer or null.
• The scheduler ranks priorities 1..4 before null/unknown values; other integers sort with null/unknown unless an implementation documents a different mapping.
• created_at and updated_at MUST represent parsed RFC 3339 instants or null; the in-memory timestamp type is implementation-defined.
• Unusable provider values for nullable fields MAY normalize to null. Unusable best-effort collection entries MAY be dropped; if no usable entries remain, use an empty list. These fallbacks MUST NOT be used for id, identifier, title, state, or explicit dispatchable.
• Preserve provider spelling in state, but trim and lowercase only for scheduler comparisons.
• blocked_by is best-effort metadata; adapters MUST NOT invent blocker semantics they cannot represent reliably.
• The GitHub Issues adapter checks native blocked_by relationships for otherwise routable open issues at polling and ID refresh. Open blockers and not_planned closures make them dispatchable=false; failed or malformed dependency reads fail closed.
• GitHub worker/intake instructions MUST register an actually required existing delivery producer as a native dependency before another unchanged blocked attempt. Attachment MUST use explicit repository/issue identity and the global issue ID, with complete paginated read-back; uncertain or duplicate mutations MUST be read back before retrying. Prose mentions are not edges or task authority. Native admission/reconciliation owns waiting and resumption of the same canonical owner, preparation and original budget; no additional continuation owner or manual hold is added. Source/evidence and unmet acceptance MUST be retained before attachment can stop a live worker. Once an open native prerequisite is verified, worker instructions MUST end the turn without a blocked marker or review handoff label, including on a final attempt, and keep the consumer open. Ordinary failure and final-attempt disposition rules apply only when there is no verified open native prerequisite. Intake MUST withhold the configured admission label from newly filed dependent consumers until attachment is verified; an already authorized consumer keeps its existing admission label. Existing retirement/no-clone policy and public read-only/private-project boundaries remain intact.
• On ID refresh, an item the provider reports as permanently deleted (GitHub 410 Gone) SHOULD be returned in a terminal state rather than omitted, so reconciliation stops its worker and cleans its workspace. An item that is merely hidden or moved (404) is omitted, which stops the worker and keeps the workspace.
• dispatchable MUST be explicit. It is true only when provider-specific eligibility checks pass; the generic scheduler never tries to reconstruct those checks from native_ref.
• native_ref MUST be null or a JSON-safe object containing only non-secret values safe to expose in prompt/tool context. If provider metadata cannot be represented safely, normalize native_ref to null; otherwise preserve the retained object verbatim.

11.4 Error Handling Contract

RECOMMENDED adapter error categories:

• unsupported_tracker_kind
• invalid_tracker_config
• missing_tracker_secret
• tracker_request (transport failure)
• tracker_status (non-success response)
• tracker_response (malformed or semantically invalid payload)
• tracker_pagination (pagination integrity failure)
• tracker_rate_limited

For portability, every adapter profile MUST document how each public language-native error form maps to a stable category and human-readable message. A literal {category, message} object is not required. Adapters MAY add retryable, retry_after_ms, provider status, and provider-specific detail, but the orchestrator only relies on success vs. failure.

Orchestrator behavior on tracker errors:

• Candidate fetch failure: log and skip dispatch for this tick.
• Running-state refresh failure: log and keep active workers running.
• Startup terminal cleanup failure: log warning and continue startup.

11.5 Tracker Writes and Agent Tools (Important Boundary)

Symphony does not require first-class tracker write APIs in the orchestrator.

• Ticket mutations (state transitions, comments, attachments, PR metadata) are typically handled by the coding agent through the selected adapter's provider-native tools.
• Tools execute in Symphony with the configured adapter credential; the child receives tool results, not a raw token.
• The current normalized issue is available to tool execution as context, including opaque native_ref, so adapters can retain provider richness without adding it to the core scheduler.
• The service remains a scheduler/runner and tracker reader.
• Workflow-specific success often means "reached the next handoff state" (for example Human Review) rather than tracker terminal state Done.

12. Prompt Construction and Context Assembly

12.1 Inputs

Inputs to prompt rendering:

• workflow.prompt_template
• normalized issue object
• OPTIONAL attempt integer (retry/continuation metadata)

12.2 Rendering Rules

• Render with strict variable checking.
• Render with strict filter checking.
• Convert issue object keys to strings for template compatibility.
• Preserve nested arrays/maps (labels, blockers) so templates can iterate.

12.3 Retry/Continuation Semantics

attempt SHOULD be passed to the template as a 1-based retry/continuation count:

• first run: attempt is null or absent;
• any later run: attempt is an integer.

The core attempt value does not distinguish a normal continuation from an error/timeout/stall retry. An implementation MAY expose an additional retry_kind template field if workflows need that distinction, but it is not part of core conformance.

12.4 Failure Semantics

If prompt rendering fails:

• Fail the run attempt immediately.
• Let the orchestrator treat it like any other worker failure and decide retry behavior.

13. Logging, Status, and Observability

Crescendo extension: retained issue delivery evidence

Issue delivery associations are retained in the existing Operations lineage and exposed read-only as usage.delivery_metrics.issue_associations (with project tags in service snapshots). The GitHub inventory task follows prospectively tracked issue workers throughout retained run history, even after completion, reconciliation or restart. Each inventory refresh rotates through at most ten retained issue scopes, so delivery reconciliation cannot sweep every historical scope each minute. It reads paginated issue cross-reference timelines and PR observations. Ownership requires an explicit closing directive or an exact standalone Symphony issue: #<number> line, a same-repository worker branch (crescendo/<number>-..., symphony/<number>-... or optionally namespaced issue-<number> with an optional suffix) and PR creation during a unique retained worker attempt. Titles and incidental references provide no ownership proof. Closing directives may start a line, an unordered Markdown bullet, or a plain prose sentence after ., ! or ?. Declarations must fit one line with at most three leading spaces; tab or deeper indentation is unsupported. Use a standalone unindented declaration for deeply nested prose. Sentence prefixes containing straight or Unicode quotation marks, backticks or > are not ownership declarations. Blockquotes, including continuation lines without another > marker, do not establish ownership. Only paragraph text can continue a quote without a marker; quoted blank lines, headings and code or a new unquoted list leave following unquoted declarations eligible. Matched backtick code spans, including multiline spans, do not establish source ownership. Heading, list and paragraph boundaries prevent unmatched backticks from hiding later prose. Backtick and tilde fenced code blocks, including those in Markdown lists or left unclosed, do not establish source ownership. Fence closers must use the opening character and at least its length, with at most three extra indentation columns beyond their container; deeper fence lines remain code. Removing examples cannot manufacture declarations. These body formats retain the same repository, issue-number, branch and worker-attempt checks. The original source/run association stays fixed across later review, merge, reopen and redispatch. Canonical handoffs and split scope are captured at worker dispatch and terminal reconciliation. Scope observed while a worker was active survives tracker edits before the first delivery observation, including removal of the handoff record or split label. Malformed handoff evidence retained from closure keeps acceptance unknown after marker removal.

The projection retains tracker observations, source and merge SHAs, evidence sources and timestamps, attempts and canonical handoff ownership. Duplicate observations are idempotent; older tracker or source timestamps cannot replace newer evidence. Completion with an owned merged source is repository_reported_completion; proven split/handoff scope is accepted_reduced_scope, never completion of the broader canonical outcome. Not-planned closure is retirement; closure without proof is unknown_acceptance. Reopen is open and preserves earlier observations and sources. These are repository-reported associations, not independently verified acceptance. Missing proof and helper usage remain explicit. Verified delivery count, cost and latency stay null, and unknown historical runs are not backfilled into accepted deliveries. Associations expire with retained worker history and respect private-project redaction. No public mutation control or local configuration change is introduced.

Canonical report-only --verify-existing delivery uses repository_reported_verification with delivery_kind: report_only and separate report_verifications; it never invents a PR source. The GitHub observer requires a completed closure with a matching final authenticated lifecycle event and one unique exact-source canonical reference. It reads bounded regular files beneath the existing private METALRAIN_SYMPHONY_EVIDENCE_ROOT: closure intent, input, acceptance, approved review, clean source, adjudicated failures, unchanged review-input digest, hosted checks and reviewer identity. Issue/repository identity, captured scope, source SHAs and reviewer work item must agree with the observed issue. Hosted commit status and check-run SHAs must match the reviewed source. All failure adjudication and unchanged-input checks remain required. Symlinked paths are rejected; ordinary receipts are limited to 128 KiB and the committed reviewer log to 16 MiB. Raw receipts, paths, review text and credentials never appear in the public projection.

Supported producer v1 review-completion.json commits the approved review and all validation/input artifact bytes, exact repository/issue/source/base/scope, original receipt/delivery attempt, reviewer run/thread/model/effort and parent worker/attempt. Its content identity uses recursively sorted compact UTF-8 JSON with integer completion values. The original usage identity projection is committed separately; accounting observation, tokens and debits are excluded. Delayed or missing usage observation cannot change completion or establish earlier approval. Legacy receipts and retained observation-only proof remain unknown without supported completion evidence.

GitHub ordering has separate clock provenance. closure-ordering.json binds that event to a new, digest-only GitHub comment and the authenticated closed/reopened lifecycle captured before review. The observer authenticates the exact comment again, including its issue, unchanged body and creation/update timestamps; missing, edited, foreign or duplicate anchors supply no proof. The current lifecycle must equal the captured prefix plus exactly one completed closure matching the issue's current state, reason and closed_at. Anchor creation must be in an earlier GitHub second than closure, and any prior lifecycle must precede the anchor. Same-second, equal, fractional or missing v1 server timestamps remain ambiguous. Local completion/usage times, file timestamps, elapsed durations and estimated clock offsets never order GitHub closure.

Operations joins the whole native review start/completion/accepted window to exactly one prospectively tracked issue worker, preserving its original run ID, item attempt and budgets. The completion must retain the original reviewer/parent identities; one reviewer run cannot own multiple completion events. Native times identify complete one-second worker buckets independently of the GitHub clock. Associations are immutable, closure-specific, idempotent across polls/restarts and expire with their original worker history. Reopen preserves history; a later closure needs its own completion and captured lifecycle. Only committed, freshly authenticated anchors demonstrably before a reopen exclude old references; incomplete references keep the current closure ambiguous.

A scope fingerprint prevents later criteria edits from inheriting earlier report acceptance. Conflicting scopes at the same GitHub timestamp remain unknown across replay/restart until a newer observation resolves scope, even when lifecycle ordering rejects the conflicting observation. The latest retained lifecycle stays intact. Lifecycle event IDs order reopen/closure observations sharing an issue update timestamp; IDs are retained only when separately read issue state and closure time match the timeline, allowing corrected reads after a race. Reusing a closure timestamp cannot reuse earlier proof, including when a reopen is missed between polls.

Dispatch/terminal handoffs, canonical ownership, original attempt budgets and unmet scope remain intact; a child report cannot accept its broader parent. Receipt polling adds no usage; auxiliary review accounting retains its separate run-ID deduplication. This is repository-reported verification, not independent acceptance. Verified cost, latency, delivery count and helper coverage stay unknown until their own proof exists. Only bounded rejection codes are projected; they refresh on repeated reads of the current tracker observation even when its timestamp and lifecycle stay unchanged. See the GH-72 investigation for the production reproduction, supported producer contract and real canonical closure verification.

13.1 Logging Conventions

REQUIRED context fields for issue-related logs:

• issue_id
• issue_identifier

REQUIRED context for coding-agent session lifecycle logs:

• session_id

Message formatting requirements:

• Use stable key=value phrasing.
• Include action outcome (completed, failed, retrying, etc.).
• Include concise failure reason when present.
• Avoid logging large raw payloads unless necessary.

13.2 Logging Outputs and Sinks

The spec does not prescribe where logs are written (stderr, file, remote sink, etc.).

Requirements:

• Operators MUST be able to see startup/validation/dispatch failures without attaching a debugger.
• Implementations MAY write to one or more sinks.
• If a configured log sink fails, the service SHOULD continue running when possible and emit an operator-visible warning through any remaining sink.

13.3 Runtime Snapshot / Monitoring Interface (OPTIONAL but RECOMMENDED)

If the implementation exposes a synchronous runtime snapshot (for dashboards or monitoring), it SHOULD return:

• running (list of running session rows)
• each running row SHOULD include turn_count
• retrying (list of retry queue rows)
• session and retry rows SHOULD include the tracker-provided issue URL when available
• codex_totals
  • input_tokens
  • output_tokens
  • total_tokens
  • seconds_running (aggregate runtime seconds as of snapshot time, including active sessions)
• rate_limits (latest coding-agent rate limit payload, if available)

RECOMMENDED snapshot error modes:

• timeout
• unavailable

For Crescendo's multi-project read-only API and dashboard, every requested project snapshot MUST retain its outcome. projects[].snapshot_status is ok, timeout, unavailable, or not_selected when excluded by the project filter. Unknown project running/ready counts are null, never zero. snapshot_status is complete or partial for the selected projects; snapshot_errors lists failed projects and their statuses. A partial response preserves observed rows with private-project redaction, sets all aggregate counts to null, and warns in health and the dashboard. Governor-held service slots remain a separate signal, not a queue estimate. A successful subsequent read clears the warning. Deployment drain and health validators MUST reject partial, filtered, or unknown project state and unknown Governor-held slot counts before treating an observation as safe.

Polling and tracker health MUST warn until the selected projects have successful tracker observations. Tracker errors make both checks critical; a selected observation older than five of its project's configured poll intervals makes both checks warn, even while a poll is in progress. A fresh successful read clears those warnings. The combined tracker observation time MUST remain unknown when any selected project has no observation or its snapshot is unavailable; otherwise it uses the oldest selected observation.

Deployment health MUST additionally require projects[].tracker_ready: true for every enabled project. The Orchestrator owns this signal: it is true only after a successful active-issue poll in the current process generation, with no subsequent poll/configuration error. Restart resets readiness even when persisted PR inventory is restored; a failed poll clears readiness until the next successful poll. Unavailable or unselected project snapshots expose null readiness. Responsive snapshots, empty queues, PR inventory timestamps and Governor-held capacity MUST NOT substitute for tracker readiness. Idle and intentionally held projects are healthy after a successful poll. Drain slot/helper accounting does not depend on tracker readiness. Startup polling uses the existing bounded health wait and unsuccessful readiness follows the existing rollback.

Service snapshots MUST also expose the Governor's current admission holds in throttle: draining covers a present dispatch drain flag, quiet-window preparation, or a quiet run finishing outside its active window. drain_reason explains that same observation: managed deployment drain is present, interrupted deployment swap awaits verified rollback, dispatch drain flag is present, preparing for quiet acceptance, or waiting for a quiet acceptance run to finish; otherwise it is null. The explicit flag takes precedence when holds overlap. Its presence does not prove a deployment is running; manual flags are independent from managed deployment state. Quiet phase MUST be observed once per policy/snapshot and reused for its drain reason so a window boundary cannot make those fields disagree. Flag contents, owner tokens and paths MUST NOT be exposed. Unavailable or legacy reasons remain unknown. research_hold is null or a map with the public project ID and phase (reserved while global research waits for idle, running while it holds the service). Expired reservations and research with project/none exclusivity MUST NOT appear as global holds. Dispatch health details show the hold, including in filtered and partial views, without inferring unknown counts or treating intentional admission policy as a health failure. These are current read-only signals, not historical attribution or dispatch controls.

Crescendo local issue workers MAY yield a prepared delivery with defer_quiet_acceptance when the existing service quiet window is configured and currently unavailable. This is bounded to one deferral per canonical logical attempt. The worker MUST commit preparation and evidence, identify its exact source commit and existing PR, provide a durable reason and retain every unmet acceptance criterion (including failed product gates). Operations MUST synchronously persist the canonical issue/attempt, final-attempt flag, PR, source, committed evidence paths and SHA-256 digests, and the wake condition before acknowledging the request. Missing persistence/evidence, stale workers, duplicate deferrals and unknown/active windows MUST fail closed. Nonignored untracked files MUST make source dirty; evidence MUST be committed regular files, not symlinks.

After acknowledgment the worker yields at its next completed turn; its run ends as deferred, without acceptance, logical failure, workspace removal or a new-budget issue/PR. The service releases its slot and continues independent work. Across worker exit and restart, ordinary tracker polling admits the same issue only in the currently configured active quiet window, under the existing canonical ownership, dependencies, quota/throttle, deployment drain and service quiet exclusivity. The original logical attempt and final-attempt disposition remain intact. Resume MUST use a fresh prompt and revalidate source and evidence identity before reuse; changed/dirty/missing source or changed/missing evidence is explicitly inapplicable. Workers MUST additionally check the current task/environment and repair product failures; deferral never passes a failed performance gate. Existing reduced-scope and retirement rules still govern unsuccessful acceptance. Read-only state exposes retained continuation identities in deferred and the scheduling reason in the waiting queue, with private-project redaction. No worker polls or sleeps on an unchanged unavailable window. Terminal or no-longer-visible issues, failed logical attempts and retired deliveries MUST clear their retained continuation durably, without resetting the canonical attempt budget. Tracker read failures MUST retain the continuation until its disposition can be observed. If a restart removes the configured quiet window, the first successful observation of an active, routable canonical issue MUST end its saved deferral through the existing failure policy. Autopilot counts the original logical attempt once and uses its existing retry/retirement limits; required quiet acceptance and failed product gates remain unmet. The removed wake condition MUST NOT park the delivery indefinitely. Replacing a dead project orchestrator MUST release its held service slots and quiet-run reservation before returning the replacement's dispatch policy, even when its exit notification has not been handled. Saved continuations and the canonical attempt budget remain intact.

Crescendo routine deployments MUST bound a busy drain (default five minutes) and leave a dispatch pause (default thirty minutes) before another busy drain, including superseding candidates. An idle service MAY bypass the pause but MUST recheck after acquiring the hold. Retry deadlines and drain observations belong to the existing deployment journal. Observations MUST report actual hold start/end and elapsed time, sampled idle time, ready work and quota/budget restrictions without attributing every unused slot to deployment. Unknown observations MUST NOT imply idle. Gate failures precede the hold. Pre-swap postponement/cancellation and verified deployment or rollback MUST release only the deployment's own hold. Interrupted swaps follow recovery below. Manual and legacy flags MUST remain independent.

Explicit managed drains MUST publish atomic owner/lifecycle state before journal appends and hold an inherited kernel lock for the operation and its swap children. A live lock MUST NOT expire from elapsed time. Governor admission checks MAY reconcile a provably abandoned pre-swap hold under that same lock, recording its lifecycle before releasing it. Dashboard reads MUST only observe. Interrupted swaps MUST remain held until the deploy path restores the previous configuration, restarts and verifies every project's health, before any unchanged-target early exit. Failed rollback, invalid ownership state or failed lifecycle journalling MUST preserve admission for later recovery. No reconciliation or manual CLI command may remove a newer owner's managed hold.

throttle.drain_owner and retained admission observations MUST expose only kind (managed, manual, legacy_unknown, quiet), phase and applicable credential-free lifecycle evidence: managed liveness (live, abandoned, recovery_required, unknown), start timestamp and commit. Owner tokens, commands and host paths MUST NOT appear. Legacy holds MUST remain unattributed and MUST NOT be automatically released. Manual CLI on/off events MUST use the shared serialized journal appender, preserving lifecycle evidence after interrupted appends. The CLI MUST report lifecycle append errors. Recovery MUST preserve active worker slots, spend, workspaces and logical attempt/retry budgets.

Under a service Governor drain, an active issue worker MUST yield at a successfully completed native turn boundary before starting another turn, after persisting any enabled checkpoint and running normal cleanup. The orchestrator MUST release its slot once and record an interrupted outcome with reason deployment_drain, retaining source/workpad, logical item attempt and retry count (including zero) without claiming failure, retirement or delivery. Existing weighted scheduling resumes the issue when the drain ends; native thread reuse is optional. Failed turns and checkpoint failures retain normal retry semantics. Standalone workflows, reconciliation, reload and shutdown retain their existing behavior. In-progress turns remain bounded by their existing execution deadlines: this cannot guarantee deployment within one five-minute drain window, and long turns or unavailable observations MUST remain explicit waits.

13.4 OPTIONAL Human-Readable Status Surface

A human-readable status surface (terminal output, dashboard, etc.) is OPTIONAL and implementation-defined.

If present, it SHOULD draw from orchestrator state/metrics only and MUST NOT be REQUIRED for correctness.

13.5 Session Metrics and Token Accounting

Token accounting rules:

• Agent events can include token counts in multiple payload shapes.
• Prefer absolute thread totals when available, such as:
  • thread/tokenUsage/updated payloads
  • total_token_usage within token-count wrapper events
• Ignore delta-style payloads such as last_token_usage for dashboard/API totals.
• Extract input/output/total token counts leniently from common field names within the selected payload.
• For absolute totals, track deltas relative to last reported totals to avoid double-counting.
• Do not treat generic usage maps as cumulative totals unless the event type defines them that way.
• Accumulate aggregate totals in orchestrator state.

Runtime accounting:

• Runtime SHOULD be reported as a live aggregate at snapshot/render time.
• Implementations MAY maintain a cumulative counter for ended sessions and add active-session elapsed time derived from running entries (for example started_at) when producing a snapshot/status view.
• Add run duration seconds to the cumulative ended-session runtime when a session ends (normal exit or cancellation/termination).
• Continuous background ticking of runtime totals is not REQUIRED.

Rate-limit tracking:

• Track the latest rate-limit payload seen in any agent update.
• Any human-readable presentation of rate-limit data is implementation-defined.

13.6 Humanized Agent Event Summaries (OPTIONAL)

Humanized summaries of raw agent protocol events are OPTIONAL.

If implemented:

• Treat them as observability-only output.
• Do not make orchestrator logic depend on humanized strings.

13.7 OPTIONAL HTTP Server Extension

This section defines an OPTIONAL HTTP interface for observability and operational control.

If implemented:

• The HTTP server is an extension and is not REQUIRED for conformance.
• The implementation MAY serve server-rendered HTML or a client-side application for the dashboard.
• The dashboard/API MUST be observability/control surfaces only and MUST NOT become REQUIRED for orchestrator correctness.

Extension config:

• server.port (integer, OPTIONAL)
  • Enables the HTTP server extension.
  • 0 requests an ephemeral port for local development and tests.
  • CLI --port overrides server.port when both are present.

Enablement (extension):

• Start the HTTP server when a CLI --port argument is provided.
• Start the HTTP server when server.port is present in WORKFLOW.md front matter.
• The server top-level key is owned by this extension.
• Positive server.port values bind that port.
• Implementations SHOULD bind loopback by default (127.0.0.1 or host equivalent) unless explicitly configured otherwise.
• Changes to HTTP listener settings (for example server.port) do not need to hot-rebind; restart-required behavior is conformant.

13.7.1 Human-Readable Dashboard (/)

Crescendo's state responses include service-wide service: {revision, commit_url} metadata, including project-filtered, full-history and snapshot-error responses. revision is the full 40-character source commit and commit_url links to that commit in ahammer/crescendo. The dashboard shows its short revision link, or "Revision unknown"; both metadata fields are null when identity is unknown.

Source deployments archive the pinned commit into releases/<sha>/source/elixir and write an empty .built marker after passing the gate. At startup one owner validates the compiled source path and this marker, then retains only public commit metadata for the process lifetime. Later main merges, environment values and workspace HEAD cannot change it. Missing, malformed or unreadable build identities, development checkouts and Burrito packages report unknown; release directories, host paths and environment values are never exposed by this metadata. This adds no configuration, control or write endpoint.

• Host a human-readable dashboard at /.
• The returned document SHOULD depict the current state of the system (for example active sessions, retry delays, token consumption, runtime totals, recent events, and health/error indicators).
• It is up to the implementation whether this is server-generated HTML or a client-side app that consumes the JSON API below.
• The dashboard MAY show each running agent's transcript: its messages, reasoning summaries, commands with output, file edits with diffs, plan steps, and images it viewed, rebuilt from the coding agent's protocol events. Transcripts SHOULD be bounded in memory (entries per run and bytes per field) and SHOULD reference images through the implementation's own artifact store (Section 13.7.3), never through workspace or host filesystem paths.
• An implementation MAY give each running agent its own page (for example /agents/<issue_identifier>) for the transcript, keeping the dashboard itself to a summary per agent.
• The agent inspector omits the configured labels.prefix and labels starting with <prefix>: before showing up to eight work labels. It uses the inspected agent's project configuration, or the single workflow's configuration, with symphony as the default prefix. An open inspector retains its last resolved prefix if configuration becomes unavailable.
• Transcripts expose raw agent output. When the dashboard is reachable beyond the operator's machine, anyone with its URL can read everything agents print or look at.

13.7.2 JSON REST API (/api/v1/*)

Provide a JSON REST API under /api/v1/* for current runtime state and operational debugging.

Minimum endpoints:

• GET /api/v1/state

  • Returns a summary view of the current system state (running sessions, retry queue/delays, aggregate token/runtime totals, latest rate limits, and any additional tracked summary fields).

  • ?history=full returns the retained Operations history: up to 2,000 lifecycle events per selected project and 48 hours of five-minute samples. It composes with project=<id>; service aggregation MUST preserve each selected project's retained events and privacy policy. The default summary and dashboard use the newest 100 events and 12 hours of samples. History is bounded, not a guaranteed cycle or lifetime record; callers MUST check timestamps before treating a window as complete. Daily totals and 14-day task averages keep their existing windows.

    Five-minute samples retain a credential-free Governor admission observation: source: governor, scope: service, ISO8601 observed_at, shared slots, Governor-held busy, draining, drain_reason, and research_hold (existing public project ID and reserved/running phase, or null). Project/none research MUST NOT appear as a global hold; unavailable and legacy observations MUST remain null. Legacy drain reasons MUST NOT be inferred or backfilled. Reasons describe the observed admission predicate, not drain ownership, intended duration or historical causality. Service samples MUST select one whole latest observation per bucket with observed_by provenance, never sum shared capacity or join different observation times. project_samples preserves each selected project's counts and timestamps; aggregate counts have counts_scope: selected_projects. Missing selected-project data makes numeric aggregates null (sample_status: partial); sparklines omit partial buckets. Bucket at is not an observation time. These sampled observations MUST NOT imply exact lost execution time, causality or historical backfill. The existing durable 48-hour retention and public/private filtering apply across restart. No task details, secrets or paths are published; the dashboard/API and admission policy remain unchanged.

  • Suggested response shape:

    {
      "generated_at": "2026-02-24T20:15:30Z",
      "counts": {
        "running": 2,
        "retrying": 1
      },
      "running": [
        {
          "issue_id": "abc123",
          "issue_identifier": "MT-649",
          "issue_url": "https://tracker.example/issues/MT-649",
          "state": "In Progress",
          "session_id": "thread-1-turn-1",
          "turn_count": 7,
          "last_event": "turn_completed",
          "last_message": "",
          "started_at": "2026-02-24T20:10:12Z",
          "last_event_at": "2026-02-24T20:14:59Z",
          "tokens": {
            "input_tokens": 1200,
            "output_tokens": 800,
            "total_tokens": 2000
          }
        }
      ],
      "retrying": [
        {
          "issue_id": "def456",
          "issue_identifier": "MT-650",
          "issue_url": "https://tracker.example/issues/MT-650",
          "attempt": 3,
          "due_at": "2026-02-24T20:16:00Z",
          "error": "no available orchestrator slots"
        }
      ],
      "codex_totals": {
        "input_tokens": 5000,
        "output_tokens": 2400,
        "total_tokens": 7400,
        "seconds_running": 1834.2
      },
      "rate_limits": null
    }

• GET /api/v1/<issue_identifier>

  • Returns issue-specific runtime/debug details for the identified issue, including any information the implementation tracks that is useful for debugging.

  • Suggested response shape:

    {
      "issue_identifier": "MT-649",
      "issue_id": "abc123",
      "status": "running",
      "workspace": {
        "path": "/tmp/symphony_workspaces/MT-649"
      },
      "attempts": {
        "restart_count": 1,
        "current_retry_attempt": 2
      },
      "running": {
        "session_id": "thread-1-turn-1",
        "turn_count": 7,
        "state": "In Progress",
        "started_at": "2026-02-24T20:10:12Z",
        "last_event": "notification",
        "last_message": "Working on tests",
        "last_event_at": "2026-02-24T20:14:59Z",
        "tokens": {
          "input_tokens": 1200,
          "output_tokens": 800,
          "total_tokens": 2000
        }
      },
      "retry": null,
      "logs": {
        "codex_session_logs": [
          {
            "label": "latest",
            "path": "/var/log/symphony/codex/MT-649/latest.log",
            "url": null
          }
        ]
      },
      "recent_events": [
        {
          "at": "2026-02-24T20:14:59Z",
          "event": "notification",
          "message": "Working on tests"
        }
      ],
      "last_error": null,
      "tracked": {}
    }

  • If the issue is unknown to the current in-memory state, return 404 with an error response (for example {\"error\":{\"code\":\"issue_not_found\",\"message\":\"...\"}}).

  • Implementations that keep transcripts MAY include the running item's transcript here (for example as transcript and workspace_summary fields) while keeping /api/v1/state light.

• POST /api/v1/refresh

  • Queues an immediate tracker poll + reconciliation cycle (best-effort trigger; implementations MAY coalesce repeated requests).

  • Suggested request body: empty body or {}.

  • Suggested response (202 Accepted) shape:

    {
      "queued": true,
      "coalesced": false,
      "requested_at": "2026-02-24T20:15:30Z",
      "operations": ["poll", "reconcile"]
    }

API design notes:

• The JSON shapes above are the RECOMMENDED baseline for interoperability and debugging ergonomics.
• Implementations MAY add fields, but SHOULD avoid breaking existing fields within a version.
• Endpoints SHOULD be read-only except for operational triggers like /refresh.
• Unsupported methods on defined routes SHOULD return 405 Method Not Allowed.
• API errors SHOULD use a JSON envelope such as {"error":{"code":"...","message":"..."}}.
• If the dashboard is a client-side app, it SHOULD consume this API rather than duplicating state logic.

13.7.3 Run Artifacts (/artifacts/<run_id>/<name>) (OPTIONAL)

Implementations that show images in transcripts MAY serve them from a run artifact store.

• Only files the implementation wrote into its artifact store are servable. Both path segments MUST match the store's own naming (for example a fixed-length hex run id and <n>.<ext>), so no request can name any other file.
• The store SHOULD accept only known image formats identified by their content rather than a claimed type, SHOULD cap the size and number of images per run, and SHOULD delete a run's images after a retention window (reference: three days after the run was last active).

14. Failure Model and Recovery Strategy

14.1 Failure Classes

1. Workflow/Config Failures

   • Missing WORKFLOW.md
   • Invalid YAML front matter
   • Unsupported tracker kind or invalid adapter-owned tracker configuration
   • Missing coding-agent executable

2. Workspace Failures

   • Workspace directory creation failure
   • Workspace population/synchronization failure (implementation-defined; can come from hooks)
   • Invalid workspace path configuration
   • Hook timeout/failure

3. Agent Session Failures

   • Startup handshake failure
   • Turn failed/cancelled
   • Turn timeout
   • User input requested and handled as failure by the implementation's documented policy
   • Subprocess exit
   • Stalled session (no activity)

4. Tracker Failures

   • Provider transport errors
   • Non-success provider responses
   • Provider-reported application errors
   • Malformed payloads

5. Observability Failures

   • Snapshot timeout
   • Dashboard render errors
   • Log sink configuration failure

14.2 Recovery Behavior

• Dispatch validation failures:

  • Skip new dispatches.
  • Keep service alive.
  • Continue reconciliation where possible.

• Worker failures:

  • Convert to retries with exponential backoff.

• Tracker candidate-fetch failures:

  • Skip this tick.
  • Try again on next tick.

• Reconciliation state-refresh failures:

  • Keep current workers.
  • Retry on next tick.

• Dashboard/log failures:

  • Do not crash the orchestrator.

14.3 Partial State Recovery (Restart)

Current design is intentionally in-memory for scheduler state. Restart recovery means the service can resume useful operation by polling tracker state and reusing preserved workspaces. It does not mean retry timers, running sessions, or live worker state survive process restart.

After restart:

• No retry timers are restored from prior process memory.
• No running sessions are assumed recoverable.
• Service recovers by:
  • startup terminal workspace cleanup
  • fresh polling of active issues
  • re-dispatching eligible work

14.4 Operator Intervention Points

Operators can control behavior by:

• Editing WORKFLOW.md (prompt and most runtime settings).
• WORKFLOW.md changes are detected and re-applied automatically without restart according to Section 6.2.
• Changing issue states in the tracker:
  • terminal state -> running session is stopped and workspace cleaned when reconciled
  • non-active state -> running session is stopped without cleanup
• Restarting the service for process recovery or deployment (not as the normal path for applying workflow config changes).

15. Security and Operational Safety

15.1 Trust Boundary Assumption

Each implementation defines its own trust boundary.

Operational safety requirements:

• Implementations SHOULD state clearly whether they are intended for trusted environments, more restrictive environments, or both.
• Implementations SHOULD state clearly whether they rely on auto-approved actions, operator approvals, stricter sandboxing, or some combination of those controls.
• Workspace isolation and path validation are important baseline controls, but they are not a substitute for whatever approval and sandbox policy an implementation chooses.

15.2 Filesystem Safety Requirements

Mandatory:

• Workspace path MUST remain under configured workspace root.
• Coding-agent cwd MUST be the per-issue workspace path for the current run.
• Workspace directory names MUST use sanitized identifiers.

RECOMMENDED additional hardening for ports:

• Run under a dedicated OS user.
• Restrict workspace root permissions.
• Mount workspace root on a dedicated volume if possible.

15.3 Secret Handling

• Support $VAR indirection in workflow config.
• Do not log API tokens or secret env values.
• Validate presence of secrets without printing them.
• Execute provider-native tracker tools in the Symphony host process with the configured adapter credential.
• Do not pass tracker credentials through the coding-agent child environment. Adapters MUST declare secret environment names so local and remote launchers can remove them from child environments.
• Do not place literal tracker credentials in a repo-owned WORKFLOW.md when the child can read that workspace; use host-side secret references instead.

15.4 Hook Script Safety

Workspace hooks are arbitrary shell scripts from WORKFLOW.md.

Implications:

• Hooks are fully trusted configuration.
• Hooks run inside the workspace directory.
• Hook output SHOULD be truncated in logs.
• Hook timeouts are REQUIRED to avoid hanging the orchestrator.

15.5 Harness Hardening Guidance

Running Codex agents against repositories, issue trackers, and other inputs that can contain sensitive data or externally-controlled content can be dangerous. A permissive deployment can lead to data leaks, destructive mutations, or full machine compromise if the agent is induced to execute harmful commands or use overly-powerful integrations.

Implementations SHOULD explicitly evaluate their own risk profile and harden the execution harness where appropriate. This specification intentionally does not mandate a single hardening posture, but implementations SHOULD NOT assume that tracker data, repository contents, prompt inputs, or tool arguments are fully trustworthy just because they originate inside a normal workflow.

Possible hardening measures include:

• Tightening Codex approval and sandbox settings described elsewhere in this specification instead of running with a maximally permissive configuration.
• Adding external isolation layers such as OS/container/VM sandboxing, network restrictions, or separate credentials beyond the built-in Codex policy controls.
• Filtering which issues, projects, boards, teams, labels, or other tracker sources are eligible for dispatch so untrusted or out-of-scope tasks do not automatically reach the agent.
• Narrowing provider-native tools so they can only read or mutate data inside the intended tracker scope, rather than exposing general workspace-wide tracker access.
• Reducing the set of client-side tools, credentials, filesystem paths, and network destinations available to the agent to the minimum needed for the workflow.

The correct controls are deployment-specific, but implementations SHOULD document them clearly and treat harness hardening as part of the core safety model rather than an optional afterthought.

16. Reference Algorithms (Language-Agnostic)

16.1 Service Startup

function start_service():
  configure_logging()
  start_observability_outputs()
  start_workflow_watch(on_change=reload_and_reapply_workflow)

  state = {
    poll_interval_ms: get_config_poll_interval_ms(),
    max_concurrent_agents: get_config_max_concurrent_agents(),
    running: {},
    claimed: set(),
    retry_attempts: {},
    completed: set(),
    codex_totals: {input_tokens: 0, output_tokens: 0, total_tokens: 0, seconds_running: 0},
    codex_rate_limits: null
  }

  validation = validate_dispatch_config()
  if validation is not ok:
    log_validation_error(validation)
    fail_startup(validation)

  startup_terminal_workspace_cleanup()
  schedule_tick(delay_ms=0)

  event_loop(state)

16.2 Poll-and-Dispatch Tick

on_tick(state):
  state = reconcile_running_issues(state)

  validation = validate_dispatch_config()
  if validation is not ok:
    log_validation_error(validation)
    notify_observers()
    schedule_tick(state.poll_interval_ms)
    return state

  issues = tracker.fetch_issues_by_states(active_states)
  if issues failed:
    log_tracker_error()
    notify_observers()
    schedule_tick(state.poll_interval_ms)
    return state

  for issue in sort_for_dispatch(issues):
    if no_available_slots(state):
      break

    if should_dispatch(issue, state):
      state = dispatch_issue(issue, state, attempt=null)

  notify_observers()
  schedule_tick(state.poll_interval_ms)
  return state

16.3 Reconcile Active Runs

function reconcile_running_issues(state):
  state = reconcile_stalled_runs(state)

  running_ids = keys(state.running)
  if running_ids is empty:
    return state

  refreshed = tracker.fetch_issues_by_ids(running_ids)
  if refreshed failed:
    log_debug("keep workers running")
    return state

  for issue in refreshed:
    if issue.state in terminal_states:
      state = terminate_running_issue(state, issue.id, cleanup_workspace=true)
    else if issue.state in active_states and issue_routable(issue):
      state.running[issue.id].issue = issue
    else:
      state = terminate_running_issue(state, issue.id, cleanup_workspace=false)

  returned_ids = set(issue.id for issue in refreshed)
  for missing_id in running_ids - returned_ids:
    state = terminate_running_issue(state, missing_id, cleanup_workspace=false)

  return state

16.4 Dispatch One Issue

function dispatch_issue(issue, state, attempt):
  worker = spawn_worker(
    fn -> run_agent_attempt(issue, attempt, parent_orchestrator_pid) end
  )

  if worker spawn failed:
    return schedule_retry(state, issue.id, next_attempt(attempt), {
      identifier: issue.identifier,
      error: "failed to spawn agent"
    })

  state.running[issue.id] = {
    worker_handle,
    monitor_handle,
    identifier: issue.identifier,
    issue,
    session_id: null,
    codex_app_server_pid: null,
    last_codex_message: null,
    last_codex_event: null,
    last_codex_timestamp: null,
    codex_input_tokens: 0,
    codex_output_tokens: 0,
    codex_total_tokens: 0,
    last_reported_input_tokens: 0,
    last_reported_output_tokens: 0,
    last_reported_total_tokens: 0,
    retry_attempt: normalize_attempt(attempt),
    started_at: now_utc()
  }

  state.claimed.add(issue.id)
  state.retry_attempts.remove(issue.id)
  return state

16.5 Worker Attempt (Workspace + Prompt + Agent)

function run_agent_attempt(issue, attempt, orchestrator_channel):
  workspace = workspace_manager.create_for_issue(issue.identifier)
  if workspace failed:
    fail_worker("workspace error")

  if run_hook("before_run", workspace.path) failed:
    fail_worker("before_run hook error")

  session = app_server.start_session(workspace=workspace.path)
  if session failed:
    run_hook_best_effort("after_run", workspace.path)
    fail_worker("agent session startup error")

  max_turns = config.agent.max_turns
  turn_number = 1

  while true:
    prompt = build_turn_prompt(workflow_template, issue, attempt, turn_number, max_turns)
    if prompt failed:
      app_server.stop_session(session)
      run_hook_best_effort("after_run", workspace.path)
      fail_worker("prompt error")

    turn_result = app_server.run_turn(
      session=session,
      prompt=prompt,
      issue=issue,
      on_message=(msg) -> send(orchestrator_channel, {codex_update, issue.id, msg})
    )

    if turn_result failed:
      app_server.stop_session(session)
      run_hook_best_effort("after_run", workspace.path)
      fail_worker("agent turn error")

    refreshed_issue = tracker.fetch_issues_by_ids([issue.id])
    if refreshed_issue failed:
      app_server.stop_session(session)
      run_hook_best_effort("after_run", workspace.path)
      fail_worker("issue state refresh error")

    if refreshed_issue is empty:
      break

    issue = refreshed_issue[0]

    if issue.state is not active or not issue_routable(issue):
      break

    if turn_number >= max_turns:
      break

    turn_number = turn_number + 1

  app_server.stop_session(session)
  run_hook_best_effort("after_run", workspace.path)

  exit_normal()

16.6 Worker Exit and Retry Handling

on_worker_exit(issue_id, reason, state):
  running_entry = state.running.remove(issue_id)
  state = add_runtime_seconds_to_totals(state, running_entry)

  if reason == normal:
    state.completed.add(issue_id)  # bookkeeping only
    state = schedule_retry(state, issue_id, 1, {
      identifier: running_entry.identifier,
      delay_type: continuation
    })
  else:
    state = schedule_retry(state, issue_id, next_attempt_from(running_entry), {
      identifier: running_entry.identifier,
      error: format("worker exited: %reason")
    })

  notify_observers()
  return state

on_retry_timer(issue_id, state):
  retry_entry = state.retry_attempts.pop(issue_id)
  if missing:
    return state

  refreshed = tracker.fetch_issues_by_ids([issue_id])
  if fetch failed:
    return schedule_retry(state, issue_id, retry_entry.attempt + 1, {
      identifier: retry_entry.identifier,
      error: "retry refresh failed"
    })

  issue = find_by_id(refreshed, issue_id)
  if issue is null:
    state.claimed.remove(issue_id)
    return state

  if not retry_dispatch_allowed(issue, state, ignore_existing_claim=issue_id):
    state.claimed.remove(issue_id)
    return state

  if no_available_slots(state):
    return schedule_retry(state, issue_id, retry_entry.attempt + 1, {
      identifier: issue.identifier,
      error: "no available orchestrator slots"
    })

  return dispatch_issue(issue, state, attempt=retry_entry.attempt)

17. Test and Validation Matrix

A conforming implementation SHOULD include tests that cover the behaviors defined in this specification.

Validation profiles:

• Core Conformance: deterministic tests REQUIRED for all conforming implementations.
• Extension Conformance: REQUIRED only for OPTIONAL features that an implementation chooses to ship.
• Real Integration Profile: environment-dependent smoke/integration checks RECOMMENDED before production use.

Unless otherwise noted, Sections 17.1 through 17.7 are Core Conformance. Bullets that begin with If ... is implemented are Extension Conformance.

17.1 Workflow and Config Parsing

• Workflow file path precedence:
  • explicit runtime path is used when provided
  • cwd default is WORKFLOW.md when no explicit runtime path is provided
• Workflow file changes are detected and trigger re-read/re-apply without restart
• Invalid workflow reload keeps last known good effective configuration and emits an operator-visible error
• Missing WORKFLOW.md returns typed error
• Invalid YAML front matter returns typed error
• Front matter non-map returns typed error
• Config defaults apply when OPTIONAL values are missing
• tracker.kind validation enforces an implementation-supported adapter
• tracker.provider preserves adapter-owned keys and validates them through the selected adapter
• $VAR resolution works for documented adapter secret keys and path values
• ~ path expansion works
• codex.command is preserved as a shell command string
• Per-state concurrency override map normalizes state names and ignores invalid values
• Prompt template renders issue and attempt
• Prompt rendering fails on unknown variables (strict mode)

17.2 Workspace Manager and Safety

• Deterministic workspace path per issue identifier
• Missing workspace directory is created
• Existing workspace directory is reused
• Existing non-directory path at workspace location is handled safely (replace or fail per implementation policy)
• OPTIONAL workspace population/synchronization errors are surfaced
• after_create hook runs only on new workspace creation
• before_run hook runs before each attempt and failure/timeouts abort the current attempt
• after_run hook runs after each attempt and failure/timeouts are logged and ignored
• before_remove hook runs on cleanup and failures/timeouts are ignored
• Workspace path sanitization, stable original-identifier-hash collision resistance, and root containment invariants are enforced before agent launch
• Identifiers unchanged by sanitization keep their deterministic workspace key; conformance tests include distinct identifiers that sanitize to the same text and verify distinct keys
• Agent launch uses the per-issue workspace path as cwd and rejects out-of-root paths

17.3 Issue Tracker Adapter

• Candidate issue fetch applies configured active states and adapter-owned scope selection
• Empty fetch_issues_by_states([]) returns empty without a provider call
• Empty fetch_issues_by_ids([]) returns empty without a provider call
• Pagination preserves order across multiple pages
• Labels are normalized to lowercase
• Unusable optional provider metadata normalizes to null/empty without hiding valid required fields
• State-list reads log omitted malformed required records; ID refresh fails malformed requested records instead of treating them as invisible
• Refresh by opaque dispatch ID returns full normalized issue snapshots
• A distinct provider ticket ID or project-item ID is preserved in native_ref when needed
• Provider-specific routing/blocker/assignment rules become explicit dispatchable
• The adapter publishes the required compact profile for config, scope, normalization, tools, and portable error mapping
• Error mapping covers config, request, non-success response, malformed payload, pagination, and rate limiting, including documented category/message mappings for language-native errors

17.4 Orchestrator Dispatch, Reconciliation, and Retry

• Dispatch sort order is priority then oldest creation time
• dispatchable=false issues are not eligible
• Required-label filtering is case-insensitive and applies after adapter normalization
• Active-state issue refresh updates running entry state
• Non-active state stops running agent without workspace cleanup
• Terminal state stops running agent and cleans workspace
• Reconciliation with no running issues is a no-op
• Normal worker exit schedules a short continuation retry (attempt 1)
• Abnormal worker exit increments retries with 10s-based exponential backoff
• Retry backoff cap uses configured agent.max_retry_backoff_ms
• Retry queue entries include attempt, due time, identifier, and error
• Stall detection kills stalled sessions and schedules retry
• Slot exhaustion requeues retries with explicit error reason
• If a snapshot API is implemented, it returns running rows, retry rows, token totals, and rate limits
• If a snapshot API is implemented, timeout/unavailable cases are surfaced

17.5 Coding-Agent App-Server Client

• Launch command uses workspace cwd and invokes bash -lc <codex.command>
• Session startup follows the targeted Codex app-server protocol.
• Client identity/capability payloads are valid when the targeted Codex app-server protocol requires them.
• Policy-related startup payloads use the implementation's documented approval/sandbox settings
• Thread and turn identities exposed by the targeted protocol are extracted and used to emit session_started
• Request/response read timeout is enforced
• Turn timeout is enforced
• Transport framing required by the targeted protocol is handled correctly
• For stdio-based transports, diagnostic stderr handling is kept separate from the protocol stream
• Command/file-change approvals are handled according to the implementation's documented policy
• Unsupported dynamic tool calls are rejected without stalling the session
• User input requests are handled according to the implementation's documented policy and do not stall indefinitely
• Usage and rate-limit telemetry exposed by the targeted protocol is extracted
• Approval, user-input-required, usage, and rate-limit signals are interpreted according to the targeted protocol
• If client-side tools are implemented, session startup advertises the supported tool specs using the targeted app-server protocol
• If provider-native agent tools are implemented:
  • only the selected adapter's tools are advertised to the session
  • valid inputs execute host-side with configured adapter auth
  • the current normalized issue and native_ref are available as internal tool context
  • tracker secrets are not inherited by the coding-agent child process
  • invalid arguments, missing auth, and transport failures return structured failure payloads
  • unsupported tool names still fail without stalling the session

17.6 Observability

• Validation failures are operator-visible
• Structured logging includes issue/session context fields
• Logging sink failures do not crash orchestration
• Token/rate-limit aggregation remains correct across repeated agent updates
• If a human-readable status surface is implemented, it is driven from orchestrator state and does not affect correctness
• If humanized event summaries are implemented, they cover key wrapper/agent event classes without changing orchestrator behavior

17.7 CLI and Host Lifecycle

• CLI accepts a positional workflow path argument (path-to-WORKFLOW.md)
• CLI uses ./WORKFLOW.md when no workflow path argument is provided
• CLI errors on nonexistent explicit workflow path or missing default ./WORKFLOW.md
• CLI surfaces startup failure cleanly
• CLI exits with success when application starts and shuts down normally
• CLI exits nonzero when startup fails or the host process exits abnormally

17.8 Real Integration Profile (RECOMMENDED)

These checks are RECOMMENDED for production readiness and MAY be skipped in CI when credentials, network access, or external service permissions are unavailable.

• A real tracker smoke test can be run with valid credentials supplied through the selected adapter's documented secret mechanism.
• Real integration tests SHOULD use isolated test identifiers/workspaces and clean up tracker artifacts when practical.
• A skipped real-integration test SHOULD be reported as skipped, not silently treated as passed.
• If a real-integration profile is explicitly enabled in CI or release validation, failures SHOULD fail that job.

18. Implementation Checklist (Definition of Done)

Use the same validation profiles as Section 17:

• Section 18.1 = Core Conformance
• Section 18.2 = Extension Conformance
• Section 18.3 = Real Integration Profile

18.1 REQUIRED for Conformance

• Workflow path selection supports explicit runtime path and cwd default
• WORKFLOW.md loader with YAML front matter + prompt body split
• Typed config layer with defaults and $ resolution
• Dynamic WORKFLOW.md watch/reload/re-apply for config and prompt
• Polling orchestrator with single-authority mutable state
• Issue tracker adapter with state-list + ID-refresh reads
• Workspace manager with sanitized, collision-resistant per-issue workspaces
• Workspace lifecycle hooks (after_create, before_run, after_run, before_remove)
• Hook timeout config (hooks.timeout_ms, default 60000)
• Coding-agent app-server subprocess client with the targeted transport/framing protocol
• Codex launch command config (codex.command, default codex app-server)
• Strict prompt rendering with issue and attempt variables
• Exponential retry queue with continuation retries after normal exit
• Configurable retry backoff cap (agent.max_retry_backoff_ms, default 5m)
• Reconciliation that stops runs on terminal/non-active tracker states
• Workspace cleanup for terminal issues (startup sweep + active transition)
• Structured logs with issue_id, issue_identifier, and session_id
• Operator-visible observability (structured logs; OPTIONAL snapshot/status surface)

18.2 RECOMMENDED Extensions (Not REQUIRED for Conformance)

• HTTP server extension honors CLI --port over server.port, uses a safe default bind host, and exposes the baseline endpoints/error semantics in Section 13.7 if shipped.
• Provider-native agent tools, when shipped, execute through the app-server session using host-side configured adapter auth without passing tracker secrets to the child.
• TODO: Persist retry queue and session metadata across process restarts.
• TODO: Make observability settings configurable in workflow front matter without prescribing UI implementation details.
• TODO: Extract common semantic helper tools only after multiple adapters demonstrate real duplication; do not preemptively replace provider-native tools with generic CRUD.

18.3 Operational Validation Before Production (RECOMMENDED)

• Run the Real Integration Profile from Section 17.8 with valid credentials and network access.
• Verify hook execution and workflow path resolution on the target host OS/shell environment.
• If the OPTIONAL HTTP server is shipped, verify the configured port behavior and loopback/default bind expectations on the target environment.

Appendix A. SSH Worker Extension (OPTIONAL)

This appendix describes a common extension profile in which Symphony keeps one central orchestrator but executes worker runs on one or more remote hosts over SSH.

Extension config:

• worker.ssh_hosts (list of SSH host strings, OPTIONAL)
  • When omitted, work runs locally.
• worker.max_concurrent_agents_per_host (positive integer, OPTIONAL)
  • Shared per-host cap applied across configured SSH hosts.

A.1 Execution Model

• The orchestrator remains the single source of truth for polling, claims, retries, and reconciliation.
• worker.ssh_hosts provides the candidate SSH destinations for remote execution.
• Each worker run is assigned to one host at a time, and that host becomes part of the run's effective execution identity along with the issue workspace.
• workspace.root is interpreted on the remote host, not on the orchestrator host.
• The coding-agent app-server is launched over SSH stdio instead of as a local subprocess, so the orchestrator still owns the session lifecycle even though commands execute remotely.
• Continuation turns inside one worker lifetime SHOULD stay on the same host and workspace.
• A remote host SHOULD satisfy the same basic contract as a local worker environment: reachable shell, writable workspace root, coding-agent executable, and any required auth or repository prerequisites.

A.2 Scheduling Notes

• SSH hosts MAY be treated as a pool for dispatch.
• Implementations MAY prefer the previously used host on retries when that host is still available.
• worker.max_concurrent_agents_per_host is an OPTIONAL shared per-host cap across configured SSH hosts.
• When all SSH hosts are at capacity, dispatch SHOULD wait rather than silently falling back to a different execution mode.
• Implementations MAY fail over to another host when the original host is unavailable before work has meaningfully started.
• Once a run has already produced side effects, a transparent rerun on another host SHOULD be treated as a new attempt, not as invisible failover.

A.3 Problems to Consider

• Remote environment drift:
  • Each host needs the expected shell environment, coding-agent executable, auth, and repository prerequisites.
• Workspace locality:
  • Workspaces are usually host-local, so moving an issue to a different host is typically a cold restart unless shared storage exists.
• Path and command safety:
  • Remote path resolution, shell quoting, and workspace-boundary checks matter more once execution crosses a machine boundary.
• Startup and failover semantics:
  • Implementations SHOULD distinguish host-connectivity/startup failures from in-workspace agent failures so the same ticket is not accidentally re-executed on multiple hosts.
• Host health and saturation:
  • A dead or overloaded host SHOULD reduce available capacity, not cause duplicate execution or an accidental fallback to local work.
• Cleanup and observability:
  • Operators need to know which host owns a run, where its workspace lives, and whether cleanup happened on the right machine.

Appendix B. Autopilot Extension (OPTIONAL)

This appendix describes an extension profile in which Symphony continuously improves one GitHub repository without a human filing work. It is implemented only for the GitHub Issues adapter.

Extension config (autopilot object):

• enabled (boolean, default false).
• channels (map name -> focus text, or name -> {focus, prompt, min_issues, max_issues, route, effort, every, when, expectations, delivers}), default cleanup, optimization, testing. Names are lowercase letters, digits, or dashes.
  • A channel object's min_issues MAY be 0 for a channel whose findings are optional.
  • every is the channel's own interval (30m, 6h, 1d, 2w; default research_cooldown_ms).
  • at (HH:MM, UTC, OPTIONAL) anchors a channel to a time of day. It first runs at the latest occurrence at or before now. After completion, use the latest anchor at or before the finish time plus ceil(every / 1d) UTC calendar days (minimum one). Thus daily tasks finishing late still leave the next day's anchor due; sub-day intervals with at run once daily, and fractional multi-day intervals round up. Delayed runs do not replay missed occurrences. Without at, the interval remains measured from completion. Failed attempts retain the 30-minute retry backoff until the attempt cap ends the run cycle.
  • A running task MUST NOT dispatch again. A task requiring at least one pull request MUST wait while an open pull request carries its channel label, including drafts, so overdue schedules and retries do not create duplicate report PRs. Closing or merging that PR releases the hold.
  • skip_unchanged: true MAY skip workspace/model creation only against a verified task input baseline younger than 24 hours. Cheap checks MUST NOT claim full evaluation coverage. Failed, short, interrupted or unverified completions MUST invalidate skip eligibility without changing their existing attempt/backoff semantics. Baselines MUST survive restart/reload. Admission MUST invalidate the old baseline durably before worker creation. The persisted contract version MUST invalidate baselines when preflight semantics change. One orchestrator owns component identity: source, complete tracker/PR inventory, evidence, delivery protection/capacity and configuration. Unknown/unreadable inputs and failures to persist or sync baseline invalidation MUST defer safely before worker creation. Evidence producers MAY opt in as a complete cohort using preflight_schema: 1 and the bounded preflight_observation bookkeeping contract in elixir/docs/planning_preflight.md. Legacy cohorts MUST retain strict identity. Resolved report pointers MAY compare substantive content at the same role; pending publication, findings, acceptance, dependencies, holds and protected delivery MUST remain identity inputs. Unknown versions/observation fields MUST defer. Retained events MUST expose bounded admission component reasons, skips and unknown-input deferrals without private paths/content. Normal-traffic canaries MUST report bounded counts, worker time and estimated cost with coverage limits, not hypothetical historical savings. Execution drain MUST remain separate from pending evaluation coverage; neither a skip nor an empty execution queue authorizes heavy checks or satisfies quiet-performance acceptance.
  • when is idle (default) or anytime.
  • effort names a rung of codex.routing.ladder for the research_route model.
  • expectations is a list of text the prompt must meet.
  • delivers is {issues: {min, max}, pull_requests: {min, max, paths}}; delivers.issues replaces min_issues/max_issues.
• repo_tasks (boolean, default true) and disabled_tasks (list): whether a repository's .crescendo/autopilot/ tasks replace channels, and which of them to skip. The folder's guidelines.md is appended to every prompt.
• min_issues_per_channel (positive integer, default 1) and max_issues_per_channel (positive integer, default 3): the issue range each research run is asked to file; min must not exceed max.
• research_route (object {model, effort}, OPTIONAL): model route for research runs, overriding label routing. Planning is the loop's hardest judgment call, so a strong model is RECOMMENDED.
• review_route (object {model, effort}, OPTIONAL): model route for pull request review runs, overriding label routing and escalation.
• max_open_issues (positive integer, default 10): research pauses while at least this many open issues carry every tracker.required_labels label.
• research_cooldown_ms (non-negative integer, default 1800000): the default interval between a channel's runs, for channels without every.
• max_pr_runs (positive integer, default 5): review runs per pull request; a pull request that reaches it without merging is retired (commented and closed).
• pr_recheck_ms (non-negative integer, default 3600000): after a review run ends without a new push, the pull request is reviewed again at the same head once this much time has passed.
• max_item_attempts (positive integer, default 3) and blocked_label (default symphony:blocked): see B.3. The blocked label is added to tracker.excluded_labels.
• trusted_associations (list, default [OWNER, MEMBER, COLLABORATOR]) and trusted_authors (list of logins, default []): pull request authors trusted to have their code run and merged.
• prompts.pull_request and prompts.research (paths relative to WORKFLOW.md, REQUIRED when enabled): prompt templates for those work-item kinds. The WORKFLOW.md body remains the issue prompt. Edits to prompt files reload like WORKFLOW.md.

B.1 Work Items

Work items carry a kind:

• issue: a tracker issue, unchanged from the core model.
• pull_request: an open GitHub pull request, returned by the adapter alongside issues with identifier PR-<number> and a pull_request object (number, head_sha, head_ref, head_repo, base_ref, draft, can_push, author, author_association, trusted, and ci_state once checked at dispatch). It is dispatchable only when it is not a draft and its author is trusted or it carries every tracker.required_labels label (a maintainer opt-in; with no required labels there is no label opt-in). required_labels is not otherwise applied to pull requests; excluded_labels is.
• research: a synthetic item per channel (id research:<channel>, state research, labels symphony:research and symphony:channel:<channel>, and a research object with channel, focus, min_issues, max_issues). Research items MUST NOT be passed to tracker reads; reconciliation skips them.

B.2 Scheduling

• Pull requests sort before issues; priority and creation time order each group.
• A pull request dispatches only when it is under max_pr_runs and either its head commit differs from the last head a review run finished at or pr_recheck_ms has passed since that run. Before spawning, the orchestrator reads CI for the head commit (commit statuses and check runs) and skips the pull request while CI is pending or unknown, so reviewers never wait on CI.
• A pull request run is a single turn sequence without continuation retries. A normal exit records the head commit it started at and when; the pull request waits for a new push or the recheck cooldown. Abnormal exits retry with backoff under agent.max_attempts. The run that reaches max_pr_runs renders with final_attempt so the reviewer merges or closes the pull request itself.
• Each channel (task) runs on its own schedule, one research run at a time.
  • A channel is due when it never ran, when every has passed since it last finished, or when its retry falls due. The most overdue due channel starts first.
  • An idle channel starts only when no agent is running and no tracker item is ready. An anytime channel starts whenever a slot is free.
  • While a research run is active nothing else dispatches (including retries), so planners' tests, headed journeys and measurements have the machine to themselves.
  • Channels that must file issues (min_issues > 0) wait while the backlog is at max_open_issues.
• When a research run ends, the implementation counts the issues and pull requests opened with the channel's label since the run started, and cleans the workspace.
  • Meeting the minimums (or counts that cannot be read) finishes the channel until it is next due.
  • Falling short, or a failed run, is a failed attempt, retried 30 minutes later.
  • The attempt that reaches max_item_attempts finishes the channel until it is next due.
• Handled heads and when they were handled, per-PR run counts, and each channel's last finish, attempts and retry time persist across restarts. A handled head persisted without a time is due for recheck.

B.3 Attempts and Retirement

Autopilot never parks work waiting for an operator.

• A failed attempt is any of: a worker ending its run with blocked_label on the issue, crash retries exhausting agent.max_attempts, or a stop for operator input or approval. The orchestrator records it (persisted), clears the label, and requeues the item; items sort behind others with fewer failed attempts.
• The run after max_item_attempts - 1 failures is the final attempt; prompts receive item_attempt and final_attempt. A final attempt should deliver, split and deliver (land the verified part and file a follow-up for the rest), or close the item as not planned.
• When an item reaches max_item_attempts failed attempts, the orchestrator retires it through the tracker: a comment with the last blocker, closing it as not planned, and closing its open draft pull requests. GitHub draft ownership requires a body line beginning with a closing keyword (close, fix, resolve, or their past/third-person forms) followed by #N, a Symphony issue: #N or Crescendo issue: #N line, or an issue-N branch segment with an optional hyphenated suffix. Matching is number-exact; incidental dependency references and non-draft pull requests MUST remain open.
• Adapters implement the optional tracker writes clear_label/2 and retire/2; an adapter without them reports unsupported_tracker_operation and the item is retried on the next poll.

B.4 Workflow Contract

The reference WORKFLOW.autopilot.md uses these conventions:

• An issue worker opens a draft pull request that closes the issue, marks it ready and labels it symphony when validated, and finally labels the issue symphony:in-review (an excluded label), which hands the work to a separate reviewer run.
• A reviewer treats pull request content as untrusted, pushes fixes only when can_push, and merges pinned to the reviewed head commit. When it abandons a pull request it closes it and releases the linked issue (closed as not planned, or relabeled symphony:needs-attention).
• symphony:hold stops and holds any issue or pull request.

B.5 Durable Outcome Facts (Crescendo)

Operations history MUST distinguish run endings, blocked item attempts and scoped item dispositions. completed measures a normal worker turn, not accepted delivery. stopped is a transport ending, not necessarily failure; reconciliation after acceptance MUST retain run time and recorded cost. New facts SHOULD retain the work item, run ID and item-attempt number. Historical attribution that was never recorded MUST remain unknown rather than inferred from nearby timestamps. Run identity MUST distinguish blocked attempts when a pull request's attempt counter resets after reopening.

A blocked attempt MUST count once independently of its run ending. Dispositions distinguish merged PR scope, verified existing work, split delivery with unmet follow-ups, retirement and unknown acceptance. Ordinary closure MUST NOT imply full acceptance. A transient PR close/reopen is a transition, not an abandoned delivery. Repeated reconciliation and replay MUST NOT duplicate accepted deliveries or attributed attempts. These facts belong in the existing durable Operations store, not a separate metrics service; counts cover its bounded retained history.

See Crescendo outcome semantics for evidence markers, retention and the state API's daily series fields.

Bounded Codex notification transport

Codex stdio input uses a 16 MiB frame assembly ceiling and a separate 4 MiB RPC/control payload ceiling. Partial turn-frame assembly has an absolute codex.read_timeout_ms deadline from its first fragment, while complete stream updates retain the configured silence timeout. Ordinary notifications drain during RPC waits; tool/approval requests use their normal handlers, and terminal/input events and startup cumulative usage remain in the bounded control queue (4 MiB / 1,024 entries). Codex 0.160.0 generated schema excerpts are checked in under elixir/test/fixtures/codex-0.160.0-output-schema.json. Diagnostic copies of supported text deltas, agent message text, command aggregatedOutput, notification command text (including commandActions copies), file-change diffs and turn-level aggregated diffs retain at most 16 KiB per field with an explicit truncation marker; raw diagnostic copies are also bounded. Only passive notification copies are shortened: executable tool/approval requests retain their original commands and arguments. RPC results, usage, model reroutes, item identity/status and terminal status are never truncated. Existing dashboard transcript entry/output limits still apply.

A frame over 16 MiB, a response or irreducible control payload over 4 MiB, or saturation of the control queue fails closed. Overflow diagnostics record the guard, observed bytes/count, phase, request identity/method, thread/turn identity and work item without payload contents. After overflow, the stream is unusable: account-usage RPC is skipped and the process group is closed during normal worker cleanup. Admitted transport overflow records an infrastructure interruption and retries after 30 seconds without advancing the implementation attempt; startup overflow retains startup admission recovery. Research interruptions use the durable channel schedule and preserve the channel's attempt count, rather than looking up synthetic IDs in the tracker. Optional native context and checkpoint reads propagate transport overflow; a failed transport rejects all later requests, including continuation turns. Native servers must split larger output into bounded delta frames; this client does not stream arbitrary JSON strings or accept unbounded controls. Output-only fields beyond the verified schema remain subject to the control payload ceiling. No billable model run is needed to verify this transport behavior.

Terminal thread usage reconciliation

Operations owns cumulative usage after worker removal. Late supported events reconcile retained run/thread/item/model/date attribution exactly once, independently of active worker actions. Terminal observations and incomplete accounting remain explicit in persisted history and summaries. Stale or duplicate notifications cannot erase previously observed turn usage evidence. Missing cache-write and separately launched helper/reviewer usage remain unknown. Run attribution expires with the existing 90-day history. A resumed thread fences its previous run's late events. Codex 0.160.0 provides cumulative token-usage notifications, not a thread/read usage snapshot. No turn may be launched solely for telemetry. See the bounded reconciliation procedure and coverage limits.

Canonical final-attempt handoffs (Crescendo extension)

Final-attempt issue prompts append the canonical handoff contract. Unchanged outcomes reuse one canonical root and its bounded attempt budget. Closed issue attempts and retirement dispositions survive polling and restarts; an issue number, diagnostic evidence or unmerged PR grants no renewal. Investigations and accepted reduced slices remain possible within the bounded policy.

GitHub admits one successor only after matching authorization records on the closed root and successor, verified merged partial delivery or a newly completed native prerequisite. Root plus native proof owns the budget; proof replay cannot reset it. Running successors stop when authorization is removed or rebound, and final-attempt throttling uses that same budget. Held or otherwise unroutable successors cannot reserve an unbound proof budget; holding an already bound successor preserves its ownership. Closed PRs still clear their review runs, failed attempts and retirement markers so reopened PRs can resume review. An exhausted open PR retries failed retirement writes on subsequent polls. Invalid and legacy unproven replacements fail admission closed without retiring their PRs or altering evidence, labels, holds or native edges. Required capabilities, dependencies and unique source/evidence must survive worker/groomer disposition. A root ends delivered, explicitly declined, or retried within policy. Local prompt rollout remains the operator's responsibility; the dashboard stays read-only. No arbitrary prose equivalence or historical migration is inferred. Legacy replacement headings prevent an issue from declaring itself a canonical root, including when its record would otherwise authorize another successor.

Helper MCP isolation (Crescendo extension)

Managed read-only helpers disable inherited MCP servers before native startup with a nested mcp_servers configuration map. Server names remain literal map keys, including punctuation; they are never encoded as quoted segments of a dotted override path. CLI map overrides use TOML inline tables, while thread configuration carries the same nested map through JSON. Existing transport definitions are preserved with enabled=false. Unknown or invalid configuration fails closed. Helpers expose only the curated source/evidence reads; lead MCP configuration is unchanged.

Operator Fast-mode default (Crescendo extension)

A local stdio MCP server owns service-wide operator controls crescendo_priority_mode_get and crescendo_priority_mode_set. It does not start an orchestrator or expose a web mutation. The setting is one atomic JSON record under the configured service state root. It contains enabled, an optional ISO 8601 expires_at with timezone, and updated_at; it survives restart. The default is standard speed. Disabling or setting a new default clears the previous expiry. Invalid writes preserve the old record; invalid or unreadable records select standard speed and report the policy error. Without an expiry, the operator default persists until changed.

Issue, PR-review, research and managed-helper threads share that default. Creation and resumption supply the native service tier, and every turn start rereads the policy to avoid cached Fast settings after disable or expiry. Fast requests use serviceTier: fast, normally acknowledged as provider tier priority; standard requests explicitly use default. Existing turns are not cancelled by preference changes. Single-workflow runtimes keep their existing native defaults. Model/effort routing, trust, helper isolation, quota and admission limits remain unchanged. Worker MCP sessions identified by their run environment cannot change the operator's setting.

The dashboard and state API remain read-only. priority_mode reports the current default and expiry; the dashboard shows On/Off, and agent badges reflect the native tier reported at thread startup. Requested speed and observed native acknowledgement remain distinct from measured latency or subscription debits. Nominal model-price USD estimates do not include tier surcharges.