Canary — deterministic regression checks for prompt changes

Publish two versions of a prompt, score the outputs you already generated for each against machine-checkable contracts, and get a CI verdict: pass, warn or block. Everything runs in this browser tab.

Scenario

Versions

Prompt name Baseline version Candidate version

Baseline template Candidate template

Contracts

Golden set (cases)

Outputs

Baseline outputs Candidate outputs Extra multi-sample baseline runs (optional flakiness evidence: a list of runs)

Policy and prices

Policy ({} = default policy)

Price in / 1K tokens Price out / 1K tokens

Pick a scenario and press Run the gate.

Score one output against one contract with the same checkers the gate uses. Every example below can be edited.

Contract type Contract Case (inputs, context, system_prompt — used by grounded, no_system_leak, must_refuse) Output Output is a JSON array of samples (needed by stable)

Pick a contract type and press Check.

What Canary is

A prompt version registry plus a deterministic regression checker. You bring the outputs your own model produced with your own keys; Canary only scores and compares them. It never calls a model, never asks for a provider key, and never uses an LLM as a judge.

Three rules the code follows

Only machine-checkable contracts are checked. JSON validity, schemas, regexes, length, PII, secrets, system-prompt leaks, refusals, grounding against a context, stability across samples. No vague "quality" score.
Blind spots are always visible. Cases without contracts, missing baseline checks and checks that could not be evaluated (skipped) are reported, never counted as a pass.
Flaky checks are detected and never used as gatekeepers. A check that flips on the same baseline build is reported, and every critical cell it would have blocked is still shown as a warn.

How this page works

This is a static Space. canary.py — byte-identical to app.py of the Gradio build — runs under Pyodide in a Web Worker in your browser. Nothing you type is sent anywhere. Each call has a time limit; if a contract's regex runs away, the worker is stopped and restarted and the result is an error, never a pass. The server build runs every user regex in a separate process with a 2 s limit instead.

The in-browser demo keeps nothing between runs and has no REST API. For the registry, golden set, flakiness history, dashboard, backups and the CI API, run the Gradio build: pip install -r requirements.txt && python app.py.

What Canary is

Three rules the code follows

How this page works

Links