DecGuardReliability testing for probabilistic AI decisions.

Define decision contracts, test behavioral invariants, detect regressions, and gate unreliable model versions before production.

Get Started

View on GitHub

$ pip install decguard
$ decguard test decguard.yaml --all

refund_request (choice)
PASS  min_accuracy   1.000 ≥ 0.9
PASS  max_ece        0.139 ≤ 0.15
PASS  min_coverage   0.727 ≥ 0.5
PASS  option_order   0 / 33 violating
PASS  whitespace     0 / 22 violating

PASS: all gates hold

Condensed output of the refund example

Decision Contracts

Define typed Choice, Noul and Score decisions once, with their gates, properties and policies, in one versioned YAML file.

Decisions and results

Reliability Testing

Measure accuracy, calibration, confidence, coverage, latency and error rate on a golden dataset, with PASS / WARN / FAIL gates.

Metrics

Metamorphic Fuzzing

Stress option order, label format, irrelevant context, whitespace, paraphrases, noul inversion and score monotonicity. Seeded, minimized, replayable.

Properties

Regression Gates

Compare two model or backend versions case by case and fail CI on answer flips, calibration loss, new errors or segment regressions.

Regression diffs

Production Checks

Analyze decision records your application already collected, offline, with baseline drift and per-segment gates.

Production checks

Backend Neutral

Run the same contract against mock, HTTP, System One (Kev, Jev through OpenRouter) or your own plugin backends.

Backends

One contract, every check

A Decision Contract describes the decision and what "reliable" means for it. Every command reads the same file.

yaml

schema_version: "0.1"

decision:
  name: refund_request
  description: How should this customer refund request be handled?
  type: choice
  options: [refund, reject, review]

backend:                   # Jev through OpenRouter; or mock, http, a plugin
  provider: systemone
  model: typesafe/jev-1.13
  url: https://openrouter.ai/api/alpha/decisions
  bearer_token_env: OPENROUTER_API_KEY

dataset: cases.jsonl

evaluation:
  probability_tolerance: 0.02   # Jev rounds probabilities to two decimals

requirements:              # violated -> FAIL, exit 1
  min_accuracy: 0.9
  max_ece: 0.1

properties:                # decguard fuzz / decguard test --all
  option_order:
    max_tv_distance: 0.03

Command	Question it answers
`decguard test`	Does the model meet its accuracy, calibration and coverage gates on the golden dataset?
`decguard fuzz`	Does the answer stay stable when irrelevant details of the input change?
`decguard diff`	Is the candidate version worse than the one it replaces?
`decguard check`	Is the deployed model still calibrated on real traffic, in every segment?
`decguard run`	Given this confidence, should the application accept, abstain, fall back or ask a human?

Exit codes are the interface: 0 pass, 1 a reliability gate failed, 2 the run could not be evaluated. Start with the quickstart, or read what DecGuard is first.