Oddish

Run evals on Harbor tasks in the cloud.

Oddish extends Harbor with:

• Provider-aware queuing and automatic retries for LLM providers
• Real-time monitoring via dashboard or CLI
• Postgres-backed state and S3 storage for logs

Just replace harbor run with oddish run.

Quick Start

1. Install

uv pip install oddish

Install latest development version

uv pip install "oddish @ git+https://github.com/abundant-ai/oddish.git#subdirectory=oddish"

2. Generate an API key here

Sign in, select an organization, and create a key from the dashboard. Organization members can create read or tasks keys; organization administrators can also create full keys. An API key cannot be used to create another API key.

export ODDISH_API_KEY="ok_..."

3. Submit a job

# Run a single agent
oddish run -d terminal-bench@2.0 -a codex -m gpt-5.5 --n-trials 3

# Or sweep multiple agents
oddish run -d terminal-bench@2.0 -c job.yaml

Example job.yaml

4. Monitor Progress

oddish status

Documentation

• CLI docs
• CLI package quick start
• Web dashboard
• Cloud backend
• Self-hosting
• Agents

License

PolyForm Noncommercial 1.0.0.

Personal and other noncommercial use is allowed under this license. It also permits use by the organizations listed in its Noncommercial Organizations section. Commercial use outside those terms requires a separate license from the rights holders. Contact the maintainer.

These terms apply to new work released under this license. They do not remove rights already granted for code released under Apache 2.0. The prior license is kept in LICENSE-APACHE-2.0. Third-party code keeps its own license. This project is no longer offered as open source under an OSI-approved license.