Quickstart

In this guide you describe a refund-handling decision, give DecGuard a handful of labeled examples, and run a reliability test against an offline mock backend. Every command and output below was produced by decguard 0.1.1 installed from PyPI.

You need DecGuard installed and an empty directory:

pip install decguard
mkdir refund-demo && cd refund-demo

1. Describe the decision

A support tool receives a customer message and must decide whether to refund, reject or send the request to manual review. Create decguard.yaml:

schema_version: "0.1"

decision:
  name: refund_request
  description: Decide how to handle a customer refund request.
  type: choice
  options: [refund, reject, review]

backend:
  provider: mock
  model: refund-mock-v1
  rules:
    - contains: "damaged"
      probabilities: {refund: 0.96, reject: 0.01, review: 0.03}
    - contains: "never arrived"
      probabilities: {refund: 0.90, reject: 0.02, review: 0.08}
    - contains: "changed my mind"
      probabilities: {refund: 0.05, reject: 0.85, review: 0.10}
  default: {refund: 0.20, reject: 0.15, review: 0.65}

dataset: cases.jsonl

evaluation:
  confidence_threshold: 0.8

requirements:
  min_accuracy: 0.9
  max_ece: 0.2

warnings:
  min_coverage: 0.6
  max_latency_p95_ms: 500

What each section says:

• decision — the question. A choice decision picks one of its options; the model returns a probability for each. description documents the question (and is what System One backends are asked).
• backend — what answers it. The mock backend is offline and deterministic: the first rule whose contains text appears in the input answers, and default answers everything else. You will swap it for a real model in step 7.
• dataset — the golden examples, relative to the contract.
• evaluation.confidence_threshold — decisions below 0.8 confidence count as abstentions: cases your application would not act on automatically.
• requirements — hard gates. If one does not hold, the run fails (exit code 1).
• warnings — soft gates. If one does not hold, the run warns (exit code 0).

2. Add a golden dataset

Create cases.jsonl, one JSON object per line. input is what the model sees; expected is the correct label.

{"id": "r1", "input": "The blender arrived damaged and won't turn on.", "expected": "refund"}
{"id": "r2", "input": "My parcel never arrived, tracking stopped two weeks ago.", "expected": "refund"}
{"id": "r3", "input": "I changed my mind about the colour, please take it back.", "expected": "reject"}
{"id": "r4", "input": "The invoice shows a different price than the website did.", "expected": "review"}
{"id": "r5", "input": "Screen was damaged in transit, photos attached.", "expected": "refund"}
{"id": "r6", "input": "I changed my mind, found it cheaper elsewhere.", "expected": "reject"}

Real datasets are larger, and may include unlabeled cases and metadata for segmentation. See Golden datasets.

3. Validate

decguard validate checks the contract, every backend's settings and the dataset, without calling any backend:

decguard validate decguard.yaml

OK  decguard.yaml
    decision  refund_request (choice): refund, reject, review
    backends  default=mock
    dataset   cases.jsonl: 6 cases, 6 labeled
    gates     requirements: 2, warnings: 2
    properties none (add 'properties' to fuzz)
    production segments: none; requirements: 0; segment requirements: 0
    policy     none
    hash      sha256:ab7ab0ad031d6130242da713be280ba3ee50dd6ad14e1665cc9a8e785ff03ecd

Validation is strict. A typo is an error that names the field, and the command exits with code 2:

error: decguard.yaml: invalid contract:
  requirements.min_acuracy: unknown field (check the spelling)

4. Test

decguard test decguard.yaml --output report.json

DecGuard 0.1.1 · refund_request (choice) · PASS

  contract  decguard.yaml  sha256:ab7ab0ad031d
  dataset   cases.jsonl  sha256:5cd7ea766986
  backend   default: mock · refund-mock-v1

  Cases        6 total · 6 decided · 0 errored · 6 labeled
  Accuracy     1.000   macro-F1 1.000
  Calibration  ECE 0.138   Brier 0.046   NLL 0.157   (10 bins)
  Selective    confidence >= 0.8: coverage 0.833   abstention 0.167   accuracy 1.000
  Latency ms   mean 0.0   p50 0.0   p95 0.0   max 0.0

  Checks
    PASS  min_accuracy                           accuracy 1 >= 0.9
    PASS  max_ece                                ece 0.1383 <= 0.2
    PASS  max_error_rate                         error_rate 0 <= 0
    PASS  min_coverage                           coverage 0.8333 >= 0.6 (warning)
    PASS  max_latency_p95_ms                     latency_p95_ms 0.007822 <= 500 (warning)

PASS: all gates hold

What DecGuard measured:

• Cases — every case got a valid answer. A backend failure would count as an error, never be dropped.
• Accuracy / macro-F1 — how often the selected label is the expected one.
• Calibration — whether confidence matches reality. The model is always right here but on average only 86% confident, so the expected calibration error (ECE) is 0.138: it is under-confident. ECE, Brier and NLL are explained in Golden tests and metrics.
• Selective — 5 of 6 decisions (0.833) clear the 0.8 threshold. The sixth, the invoice question, got review at 0.65: correct, but not confident enough to automate.
• Latency — time spent in the backend call. The mock answers in microseconds.
• Checks — one line per gate. max_error_rate was not configured: it is an implicit requirement that no case may error.

--output also wrote the complete report, including every decision, to report.json. decguard report report.json prints it again at any time.

5. Check behavior, not only answers

Golden tests check answers to fixed inputs. Metamorphic properties check that the answer does not change when it should not: here, when the options are presented in a different order, or an irrelevant sentence is added to the message. Append to decguard.yaml:

properties:
  option_order:
    max_tv_distance: 0.03
  irrelevant_context:
    max_tv_distance: 0.05

max_tv_distance is how far the probability distribution may move (total variation distance, 0–1) on any transformed case. Run the golden gates and the properties together:

decguard test decguard.yaml --all

  Properties   seed 0
    option_order         18 compared · 0 violating (0.0%) · flips 0 (0.0%) · max TV 0.000
    irrelevant_context   11 compared · 0 violating (0.0%) · flips 0 (0.0%) · max TV 0.000

  Checks
    ...
    PASS  option_order.max_error_rate            option_order.error_rate 0 <= 0
    PASS  option_order.max_violation_rate        option_order.violation_rate 0 <= 0 (per case: max_tv_distance 0.03)
    PASS  irrelevant_context.max_error_rate      irrelevant_context.error_rate 0 <= 0
    PASS  irrelevant_context.max_violation_rate  irrelevant_context.violation_rate 0 <= 0 (per case: max_tv_distance 0.05)

PASS: all gates hold

Transformations are derived from a seed, so every run checks exactly the same inputs. Fuzzing and replay shows a model that fails this check.

6. PASS, WARN and FAIL

Every run ends in one status, and the exit code follows from it:

Status	Meaning	Exit code
PASS	every requirement and every warning holds	0
WARN	every requirement holds, at least one warning does not	0 (1 with --fail-on-warn)
FAIL	at least one requirement does not hold	1
error	the run could not be evaluated (invalid contract, unreachable backend, ...)	2

A warning. Raise the coverage warning to min_coverage: 0.9 and run decguard test decguard.yaml again. Coverage is 0.833, so:

    WARN  min_coverage                           coverage 0.8333 violates >= 0.9 (warning)

WARN: requirements hold, some warning gates do not

The exit code is still 0, so a pipeline continues; add --fail-on-warn where warnings must block.

A failure. Put min_coverage back to 0.6 and add a case the mock gets wrong — the customer mentions damage but wants a replacement, which should go to review:

{"id": "r7", "input": "The lid was damaged, but I just want a replacement lid.", "expected": "review"}

  Checks
    FAIL  min_accuracy                           accuracy 0.8571 violates >= 0.9
    FAIL  max_ece                                ece 0.2329 violates <= 0.2
    PASS  max_error_rate                         error_rate 0 <= 0
    PASS  min_coverage                           coverage 0.8571 >= 0.6 (warning)
    PASS  max_latency_p95_ms                     latency_p95_ms 0.007859 <= 500 (warning)

  Failures (1 of 1)
    r7           expected review, got refund (0.96)
                 input: The lid was damaged, but I just want a replacement lid.

FAIL: at least one requirement does not hold

The model is wrong and 96% confident, so accuracy drops and calibration fails with it. The failing example is printed with its input, and the command exits 1. This is the signal a CI job gates on.

7. Connect a real model

Only the backend section changes. For example, to ask TypeSafe's Jev through OpenRouter's Decisions API:

backend:
  provider: systemone
  model: typesafe/jev-1.13
  url: https://openrouter.ai/api/alpha/decisions
  bearer_token_env: OPENROUTER_API_KEY

evaluation:
  confidence_threshold: 0.8
  probability_tolerance: 0.02   # Jev rounds probabilities to two decimals

export OPENROUTER_API_KEY=...   # the key stays in the environment, never in the contract
decguard test decguard.yaml

You can also keep the mock as the default and add real models under backends, then choose one with decguard test decguard.yaml --backend NAME. See Backends for Kev, HTTP endpoints and your own Python code.

Next steps

• Understand the building blocks: Decisions and results, Golden datasets, Requirements and warnings.
• Explore the complete refund example in the repository, examples/refund: an order-sensitive model, regression limits, production records and a routing policy.
• Find what golden tests miss with metamorphic properties and fuzzing.
• Compare model versions with regression diffs.
• Check deployed models with production checks, and route live decisions with policies.
• Gate pull requests with GitHub Actions.