Starlight

Starlight is a high-frequency workflow engine written in Rust. It runs asynchronous tasks connected by named channels, letting you build pipelines that generate, transform, route, and deliver data.

Define a workflow in YAML or JSON, then manage it through the HTTP API or CLI. Workflows form directed acyclic graphs (DAGs): tasks run concurrently and pass JSON messages downstream through buffered channels.

Development status: Starlight is still under active development. Features, configuration formats, and APIs may change as the project evolves.

What you can do

• Generate data from timers, random numbers, CSV files, or simulated signals.
• Filter, map, convert, split, and aggregate messages as they flow through tasks.
• Send results to logs, CSV files, or HTTP endpoints.
• Start, pause, resume, and stop workflows, and inspect per-task state and metrics.
• Generate workflow configurations interactively using an LLM.
• Extend the engine with custom Rust tasks.

Quick start

You need Rust and Cargo with support for the Rust 2024 edition. Run the following commands from the repository root.

1. Build and start the server

cargo build --workspace
RUST_LOG=info cargo run --bin starlight

The API listens on port 8246 by default. Leave the server running while using the CLI in another terminal.

2. Create a workflow

Save this as demo.yaml. It generates a random integer between 1 and 100 every 500 milliseconds and logs each message in the server terminal.

id: demo
name: Random Number Demo
description: Generate random numbers and log them.
channel_buffer_size: 1000
tasks:
  - id: generator
    type: number_generator
    params:
      min: 1
      max: 100
      interval_ms: 500
    dependencies: []
    outputs:
      out: [numbers]

  - id: log
    type: logger
    params:
      prefix: "[demo]"
      level: info
    dependencies: [numbers]
    outputs: {}

The generator writes to the channel numbers; the logger receives from that same channel. A source has no dependencies, and a sink has no outputs.

3. Run and control the workflow

In a second terminal, open the repository root and create a shell alias for the CLI. The executable currently builds as cli; the alias lets you use sl in the commands below.

alias sl="$PWD/target/debug/cli"

sl push demo.yaml     # Save the configuration on the server
sl run demo           # Mount the saved workflow and start it
sl list               # List loaded workflows
sl state demo         # Inspect task state and metrics

sl pause demo
sl start demo         # Resume a paused workflow
sl stop demo

You can also invoke the CLI through Cargo without an alias:

cargo run -p cli -- list

The CLI uses http://localhost:8246 when no context is selected. See Configuration to connect to another server.

Working with workflows

Workflow files and loaded workflows are managed separately:

• Push saves a configuration to disk without loading or starting it.
• Mount loads a saved configuration into the engine without starting it.
• Run mounts and starts a workflow in one command.
• Unmount unloads a workflow while keeping its saved file.
• Remove deletes the saved file and unloads the workflow if it is loaded.

Files are stored in .starlight/engine/workflows/ by default, relative to the server's working directory. Files remain on disk after a server restart; mount and start them again to run them.

Command	Purpose
sl list	List loaded workflows
sl list --all	List saved workflow files with their loaded status
sl push <file.yaml>	Save a YAML or JSON configuration on the server
sl pull <id>	Download a configuration as <id>.yaml
sl pull <id> --output json	Download a configuration as <id>.json
sl mount <id>	Load a saved workflow
sl unmount <id>	Unload a workflow and keep its file
sl run <id>	Mount and start a workflow
sl start <id>	Start, resume, or restart a loaded workflow
sl pause <id>	Pause a workflow
sl stop <id>	Stop a workflow
sl state <id>	Show per-task state and metrics
sl remove <id>	Delete a saved workflow and unload it
sl generate	Open the interactive workflow generation chat

Restarting a stopped workflow creates fresh task instances and channels, so its task state is reset. Resuming a paused workflow continues with its existing tasks.

Configuration format

Each workflow has an id, a name, and a list of tasks. The description and channel_buffer_size fields are optional; the default channel buffer size is 1000.

Each task specifies a unique id, a registered type, task-specific params, input dependencies, and outputs. Channel IDs are shared across the workflow: connect tasks by using the same channel ID in an output and a dependency. The engine supports fan-out to multiple consumers and merging multiple input channels.

A flat dependency list uses the default input port, in. Tasks that support multiple input ports can use a named mapping:

dependencies:
  left: [channel_a]
  right: [channel_b]

When a workflow is loaded, the engine checks for unknown task types, duplicate task IDs, unresolved dependencies, and cycles.

Available tasks

Task type	Purpose
number_generator	Emit random integers at a configured interval
timer	Emit a payload on an interval or cron schedule
csv_reader	Stream rows from a CSV file
csv_writer	Write messages to a CSV file
logger	Log incoming messages
filter	Pass or drop messages based on field conditions
splitter	Route messages to outputs based on field values
json_mapper	Project or rename JSON fields
type_converter	Convert field types
math_exp_eval	Evaluate expressions and add result fields
aggregator	Compute windowed count, sum, average, min, max, or collections
http_sender	Send messages to an HTTP endpoint
dummy	No-op task for testing
simulator	Generate composite signals such as sine waves, trends, and anomalies

Task parameters and examples are documented in the built-in task implementations. The simulator task is registered by the server; its signal models are described in the simulator README.

AI workflow generation

Workflow generation is optional and requires a running LLM endpoint that supports the OpenAI-compatible /chat/completions API. Configure LLM_BASE_URL and LLM_MODEL on the server, then run:

sl generate

Describe the workflow you want and answer any follow-up questions. Press Ctrl+S to save the latest generated configuration as <id>.yaml in the current directory, and Esc to leave the chat. Use sl push <id>.yaml and sl run <id> to save and run it on the server.

You can also generate a workflow through the API:

curl -X POST http://localhost:8246/workflows/generate \
  -H 'Content-Type: application/json' \
  -d '{
    "prompt": "Generate a random integer between 1 and 100 every second and log it",
    "model": "qwen3.5-9b",
    "base_url": "http://localhost:1234/v1",
    "auto_load": false
  }'

The server validates generated configurations and feeds errors back to the model for repair, with up to three total attempts. Responses have a status of questions, completed, or validation_failed. To continue a conversation, send the returned messages history with your next user message instead of prompt. Setting auto_load to true loads a valid configuration into the engine; start it separately through the workflow API.

HTTP API

The CLI uses the same API that you can call from scripts or other applications. Paths below assume the default empty BASE_PATH.

Method	Path	Purpose
GET	/health	Check server health
GET	/workflows	List loaded workflows
GET	/workflows/{id}	Get workflow information
GET	/workflows/{id}/state	Get per-task state and metrics
PATCH	/workflows/{id}	Send a start, pause, or stop command
POST	/workflows/{id}/mount	Load a saved workflow
POST	/workflows/{id}/unmount	Unload a workflow
GET	/workflows/files	List saved workflow files
POST	/workflows/files	Save a configuration as YAML; request body is JSON
GET	/workflows/files/{id}	Download a saved configuration as JSON
DELETE	/workflows/files/{id}	Delete a saved file and unload its workflow
POST	/workflows/generate	Generate a workflow with an LLM

For example, pause the demo workflow:

curl -X PATCH http://localhost:8246/workflows/demo \
  -H 'Content-Type: application/json' \
  -d '{"command": "pause"}'

Configuration

The server reads environment variables and loads a .env file when present.

Variable	Default	Purpose
PORT	8246	API server port
BASE_PATH	Empty	Prefix for all API routes, such as /api
ENGINE_DIR	.starlight/engine	Server storage directory; workflows live in its workflows/ subdirectory
LLM_MODEL	qwen3.5-9b	Default model for workflow generation
LLM_BASE_URL	http://localhost:1234/v1	Default LLM API base URL
CLI_DIR	.starlight/cli	CLI configuration directory, resolved relative to the user's home directory unless absolute
RUST_LOG	Unset	Logging filter, such as info or debug

To configure a CLI context, edit ~/.starlight/cli/config.yaml (or the file under CLI_DIR). For example:

current_context: local
contexts:
  local:
    name: local
    description: Local Starlight server
    endpoint: http://localhost:8246
    api_key: ""

List and select configured contexts with:

sl config get-contexts
sl config set-context --name local

If the server uses a different port or BASE_PATH, include it in the context's endpoint, for example http://localhost:9000/api.

Development

Starlight is a Cargo workspace with an API server and four crates:

src/                Axum API server and custom simulator task
crates/eng/         Workflow engine, task lifecycle, channels, and built-in tasks
crates/cli/         Command-line client and interactive generation TUI
crates/simulator/   Statistical signal models
crates/jb/          JSON builder utility

Build and run tests across the workspace:

cargo build --workspace
cargo test --workspace
cargo test -p eng

Custom tasks implement the Task trait and expose a factory that accepts a task ID and JSON parameters. Register the factory with engine.register_task("my_type", MyTask::create) before loading workflows. Use ctx.running().await as the execution loop guard so tasks respond to pause and stop commands. See the simulator task for an example.

License

Starlight is source-available under the Elastic License 2.0 (ELv2). The license allows use, modification, and redistribution subject to its terms. It prohibits providing Starlight to third parties as a hosted or managed service where users have access to any substantial set of its features or functionality. See LICENSE for the full terms.