knock knock
Open up. It's the taxman — you're being audited.
A command-line tool for auditing AI providers: send a file of messages to a model and record every raw response. Also usable as a Python package.
uv tool install "ai-taxman[all]" # every provider
#uv tool install "ai-taxman[openai]" # one provider
#uv tool install "ai-taxman[openai,anthropic]" # severalCurrently available: openai, gemini, and anthropic.
taxman init # set up a project in this directory
taxman audits new openai <audit-name> # writes taxman/audits/<audit-name>.yamlEdit the audit file: set api_key_env: to the name of the variable holding your key, and change any other setting.
Then write your messages, one per line:
export OPENAI_API_KEY="sk-..."
echo "When is the next US federal election?" > taxman/messages/<audit-name>.txt
taxman audits validate <audit-name> # check everything; sends nothing
taxman collect <audit-name> # sends the messages; bills your accountResponses are written to taxman/data/<audit-name>/<run_id>/responses.jsonl.
- Set up a project.
taxman initcreates ataxmanproject, which contains a handful of directories. - Create an audit.
taxman audits new <provider> <audit-name>creates an "audit file" (<audit-name>.yaml) for the user-specified provider (<provider>). All audit settings and parameters are specified here with human-readable text. - Write messages. A
.txt"messages file" contains all of the messages that will be sent (separately) to the AI provider. One message per line. - Add a system prompt (optional). A system prompt, sent with every message, can be specified in a separate
.txtfile. - Collect.
taxman collect <audit-name>collects the audit as specified in the audit file and creates a manifest of what was run.
A project is a directory with a taxman.yaml at its root.
taxman init creates a project in the current working directory and prompts the user to set paths manually or accept the defaults.
If taxman init --yes is called, default paths are automatically accepted.
taxman.yaml # marks the root and records the folders below
taxman/
├── audits/ # one YAML file per audit
├── messages/ # .txt messages files
├── prompts/ # system prompt files
├── data/ # collected responses
└── logs/ # run logsCommands work from anywhere inside the project, like git.
taxman audits new <provider> <audit-name> takes no other arguments.
It writes every setting at its default, with a comment above each giving its type, default, options, and a link to the provider's docs.
You configure an audit by editing that file.
Two settings are placeholders that a run refuses until you fill them in: api_key_env:, and
the model's name, whose comment links the provider's list of current models.
Without the comments:
audit: <audit-name>
provider: openai
api_key_env: <insert_api_key_env_var_here>
messages: taxman/messages/<audit-name>.txt
system_prompt:
output:
dir: taxman/data/{audit}/{run_id}
filename: responses.jsonl
compress: false
log_dir: taxman/logs/{audit}
execution:
repeats: 1 # times each message is sent
max_concurrency: 4
batch: false
timeout_s: 120
max_retries: 5
on_error: continue
model: # provider-specific; this is OpenAI's
name: <insert_model_name_here>
temperature:
# ...Relative paths resolve against the project root.
api_key_env: names one environment variable, in capitals (OPENAI_API_KEY), and taxman
reads only that one. It never stores a key. Export the variable however you like (shell profile, direnv, CI secret).
Because each audit names its own variable, one machine can hold several keys for the same
provider.
If the field is missing, still the placeholder, or names an unset variable, collect stops
before sending anything.
One message per line in a .txt file. Blank lines and lines starting with # are skipped;
start a line with \# to send a message that begins with #.
Each message may appear only once; to send one more than once, set execution.repeats.
Put the text in a file and name it in the audit:
system_prompt: taxman/prompts/neutral.txtEach provider sends it through its own API field. The manifest records the file's path and text.
taxman collect <audit-name>Each message is sent repeats times, concurrently. Every setting comes from the audit file;
there are no command-line overrides. Each run writes to its own directory:
responses.jsonl: one row per response, flushed as it arrives.manifest.json: resolved config, tool version, message ids, counts, and timings. It is written before the first request and updated when the run ends.
The manifest's status is running until the run ends, then complete, stopped_early,
interrupted, or failed. n_messages × repeats is the number of responses expected. A
killed run keeps every response already received.
taxman collect <audit-name> on an audit already collected finishes its latest run, in the same
directory. It sends only what that run is missing: responses never received, and responses that
failed. It says so before it starts.
- If the run already has every response, nothing is sent; it says the audit is complete.
- If the audit has changed since the run began, it refuses, and lists what changed. Every
setting counts, as do the messages and the system prompt text. Reordering lines or editing
#comments does not count. - If another
collectis still collecting the run, it refuses rather than send the same messages twice.
--new-run starts a new run beside the old ones, whatever state the latest is in. That needs
{run_id} in output.dir, which taxman audits new writes. Without it, the folder holds one
run: move that folder elsewhere and collect again to start over.
| Option | Effect |
|---|---|
--new-run |
Start a new run, instead of finishing the latest one |
Each run writes its log to <output.log_dir>/<run_id>.log, and only there: one line per
response, every retry and why, and how the run ended. collect prints the log's path when it
starts, then shows a progress bar on stderr while it runs; the bar is left out when stderr
is not a terminal. Message text and API keys are never logged; the log does name the variable
the key was read from.
A request that fails with a rate limit, a timeout, or a server error is retried, up to
execution.max_retries times. When the provider says how long to wait (a Retry-After
header, which OpenAI and Anthropic send), every request pauses for that long, not just the
one that was told, and the progress bar and the log say when it resumes. Otherwise each retry
waits a little longer than the last, up to 30 seconds. Every wait also gets a random extra of
up to a second, so the requests it held back do not all go out at once.
A wait over 5 minutes, or a used-up quota (Gemini's daily quota_exceeded and spent prepaid
credit, OpenAI's billing and spend limits, Anthropic's monthly spend cap, spend limits and
billing errors), fails the request and stops the run, since anything else sent meanwhile
would fail the same way; collecting again later finishes it.
kill <pid> stops a run gracefully: in-flight requests finish and the manifest is set to
stopped_early. A request waiting to retry is not sent again, but recorded as failed.
Collecting the audit again finishes the run. After kill -9 the responses on
disk survive, but the manifest still says running.
While a run is being collected, the pid of the process collecting it is in collect.pid in
the run directory, so kill $(cat <run dir>/collect.pid) stops it. A run ended by kill -9
cannot remove its collect.pid, so the file names a dead process until the run is collected
again.
taxman has no background mode of its own; the shell's works. To collect several audits at once:
taxman collect openai-probe > openai.txt 2>&1 &
taxman collect anthropic-probe > anthropic.txt 2>&1 &
waitOne JSON object per line, the same shape for every provider:
{
"schema_version": 1,
"audit": "<audit-name>",
"run_id": "20260830T102201-a1b2c3",
"message_id": "4b841e8b-62f2-5814-b265-36aeb267e4b0",
"message": "When is the next US federal election?",
"repeat": 2,
"provider": "openai",
"model": "gpt-5",
"requested_at": "2026-08-30T10:22:01.004512-04:00",
"received_at": "2026-08-30T10:22:01.816730-04:00",
"latency_ms": 812,
"status": "ok",
"error": null,
"attempts": 1,
"raw": {}
}raw is the provider's response, verbatim. taxman does not parse it; extracting text,
token counts, or citations is a separate step run on the data afterward.
requested_at is when the request was first sent, and latency_ms runs from then to the
last answer, the waits between retries included.
Every time taxman writes, in rows, manifests and logs, is your local time with its offset from UTC, so each stays exact if you compare across time zones.
message_id is a UUID5 of the message's text, so the same message has the same id in every
file and every run. An id can be computed from text, but never turned back into it.
taxman messages ids taxman/messages/probe.txt # each message in a file with its id, as CSV
taxman messages ids -t "Hello" -t "Goodbye" # the same, for messages typed here
taxman messages text <id> taxman/messages/probe.txt # the message in a file with this idIn Python, from ai_taxman import message_id gives the id of a string.
Outside taxman, an id is uuid5(uuid5(NAMESPACE_URL, "https://matthewdeverna.com/"), text),
where text is the line with surrounding whitespace removed. The trailing / is part of the
namespace.
Changes are additive unless listed here.
| Version | Change |
|---|---|
| 1 | Initial schema. |
| Command | What it does |
|---|---|
taxman init |
Set up a project in this directory |
taxman audits new <provider> <audit-name> |
Write a new audit file |
taxman audits list |
List the project's audits |
taxman audits show <audit-name> |
Print an audit's resolved settings |
taxman audits validate <audit-name> |
Check an audit, and say what collect would do, without sending anything |
taxman collect <audit-name> |
Run an audit |
taxman messages ids <file> |
Print each message in a file with its id |
taxman messages ids -t <message>... |
Print the id of each message given |
taxman messages text <id> <file> |
Print the message in a file with this id |
taxman providers |
List the installed providers |
taxman doctor |
Check PATH and tab completion, and offer to fix them |
from ai_taxman import run_audit
result = run_audit("<audit-name>")
print(result.n_ok, result.n_error, result.output_path)A name resolves as on the command line; a path ending in .yaml is read directly.
run_audit also accepts an AuditConfig from load_audit() and an on_record callback.
Like collect, it finishes the audit's latest run; new_run=True starts another, and
result.resumed and result.already_complete say which happened. n_ok and n_error
count what this call sent; run_n_ok and run_n_error count the whole run.
taxman --install-completion # or accept the offer from `taxman doctor`Restart your shell. Completion covers commands, providers, and audit names.
zsh: check startup time. The installer appends
fpath+=~/.zfunc; autoload -Uz compinit; compinit to ~/.zshrc. If your setup already runs
compinit (oh-my-zsh, prezto, most prompts), zsh then rebuilds its completion cache on every
start. Run time zsh -i -c exit twice; if the second run takes much more than 0.2s:
- Shorten the installer's line to
fpath+=~/.zfunc. - Move it to the top of
~/.zshrc, above anysourceline.compinitreadsfpathonce, so a directory added after it is ignored. rm ~/.zcompdumpand open a new terminal.
Check with print -r -- ${_comps[taxman]:-NOT REGISTERED}. It should print _taxman; if it
prints NOT REGISTERED, the fpath line is still below compinit.
The installer also adds zstyle ':completion:*' menu select, which enables arrow-key menus
for every command. Delete it if you don't want that.
To uninstall: rm ~/.zfunc/_taxman and delete the block in ~/.zshrc marked
# taxman additions. bash and fish need no extra steps.
| Provider | Extra | Status |
|---|---|---|
| OpenAI | ai-taxman[openai] |
Responses API; no batch mode yet |
| Anthropic | ai-taxman[anthropic] |
Messages API; no batch mode yet |
| Google Gemini | ai-taxman[gemini] |
Interactions API; no batch mode or safety settings |
Each provider is isolated from the others. To add one, see CLAUDE.md and
docs/provider-apis/.
uv sync --all-extras
uv run pytestMIT — see LICENSE. No tax is owed on this one.