Two minutes. One command. Your own key.

Is your endpoint serving the model you pay for?
Check it now.

Send 28 fixed probes three times to your endpoint and compare its behavior with a public Martingale Bench reference. Responses stay on your machine. References are downloaded separately; optional Slack notifications share only result metadata.

Set up your key and Slack

Backed by Y Combinator

$ npx --yes --package https://martingale.io/assets/cli/martingale-check-0.2.2.tgz martingale-check --model gpt-4.1

Recorded example · October 15, 2026

Reference: gpt-4.1, 40 runs, 1,120 responses
Endpoint: api.openai.com · model "gpt-4.1" · 84 calls

CONSISTENT   p = 0.0703 (threshold 0.01)
No detectable difference at this sample size.
This does not establish model identity or equivalence.

Node.js 20+. Reads OPENAI_API_KEY from your environment. Provider charges apply; retries can add calls. Your result may differ from this recorded example. Set up your key and Slack

As seen on

USA TODAY Mirror-Democrat The Daily Tribune News Lincoln Journal

The full stack for understanding AI drift.

Two complementary ways to see what changed: the tasks a model can do, and the behavior that makes it recognizable.

Task-based detection

Canary.

Keep a constant set of tasks in view. Canary runs repeatable checks and compares the results over time, so changes in correctness, format and tool use become measurable.

Explore the public bench

Behavioral fingerprints

Raven Alpha.

A model can change while still passing the test. Raven Alpha compares response patterns to fixed prompts, looking for shifts in behavior that a pass rate can miss.

How it works

Compare responses over time.

Repeat the same probes. Compare with the baseline. Bring the evidence into one investigation.

Observe
Repeatable tasks and fixed prompts.
Compare
Measure the change against a reference.
Investigate
Follow the signal back to its evidence.

Studying how models change.

We’re building a clearer picture of how AI changes after deployment. That means studying both capability regressions and quieter shifts in behavior, then testing which signals hold up.

Our work brings together controlled model comparisons, repeatable probes and a public bench. We publish the methods and their limits so the evidence can be examined, challenged and improved.

Run the check

Check your endpoint.
Get the result in Slack.

Start with GPT-4.1, then use your own compatible endpoint. The checker runs in your terminal with your provider account. You choose whether to share the result with your team.

84 planned requests per direct check. Provider charges apply; retries can add calls and completion time varies.
Enter keys in your terminal, never this page. Your provider key is sent to the endpoint you choose.
Slack receives the result, model names and a short description of measured differences. Detailed statistics stay in your terminal. No prompts, raw answers or keys.

This free check uses Raven Alpha fingerprint detection. It compares response patterns; Canary's pass/fail checks are a separate capability. CONSISTENT means no detectable difference at this sample size. MISMATCH does not establish the cause or a quality loss. Errors and insufficient evidence are not reported as successful checks.

Download 0.2.2 Available references

The commands download version 0.2.2 directly from this website. No global installation is needed.

Set your provider key

Install Node.js 20 or newer, then open a bash or zsh terminal. Paste this command and press Return once. When asked for your API key, paste the key and press Return. Nothing appears while you paste; this is expected.
terminal · private prompt
```
unset OPENAI_API_KEY; while true; do printf 'OpenAI API key: '; if ! read -r -s OPENAI_API_KEY; then unset OPENAI_API_KEY; printf '\nSetup cancelled.\n'; break; fi; printf '\n'; if printf '%s' "$OPENAI_API_KEY" | LC_ALL=C grep -Eq '^[^[:space:][:cntrl:]]+$'; then export OPENAI_API_KEY; printf 'API key saved. Continue to step 2.\n'; break; fi; unset OPENAI_API_KEY; printf 'No key entered, or the paste contains extra characters. Paste only your key, then press Return.\n'; done
```
Wait for API key saved, then continue in the same terminal. This saves the key for this terminal session; the actual check starts in step 3. Secrets stay out of command history. Press Control-C to cancel a prompt.

Connect your Slack optional

Create an incoming webhook in Slack and choose your channel. Run this command, then paste only the webhook URL at the hidden prompt and press Return. An empty or malformed entry asks you to try again.

terminal · private prompt

unset MARTINGALE_SLACK_WEBHOOK; while true; do printf 'Slack webhook URL: '; if ! read -r -s MARTINGALE_SLACK_WEBHOOK; then unset MARTINGALE_SLACK_WEBHOOK; printf '\nSetup cancelled.\n'; break; fi; printf '\n'; if printf '%s' "$MARTINGALE_SLACK_WEBHOOK" | LC_ALL=C grep -Eq '^https://hooks[.]slack(-gov)?[.]com/services/[A-Za-z0-9_-]+/[A-Za-z0-9_-]+/[A-Za-z0-9_-]+$'; then export MARTINGALE_SLACK_WEBHOOK; printf 'Slack webhook saved. Continue to step 3.\n'; break; fi; unset MARTINGALE_SLACK_WEBHOOK; printf 'No valid Slack webhook entered. Paste only the full https://hooks.slack.com/services/... URL, then press Return.\n'; done

Wait for Slack webhook saved before step 3. This checks the URL format; delivery is confirmed separately after the check finishes.

Send this result to Slack

Copy and run the command below. It downloads the checker, makes fresh endpoint requests and sends the measured result to your channel.
terminal · check + Slack
```
npx --yes --package https://martingale.io/assets/cli/martingale-check-0.2.2.tgz martingale-check --model gpt-4.1 --slack-webhook-env MARTINGALE_SLACK_WEBHOOK --slack-on all
```
--slack-on all sends every completed result. Slack says “No mismatch detected” or “Mismatch detected,” with a short description of measured differences. Remove --slack-on all to notify only on mismatches. No message is promised if the check cannot complete; inspect Slack before retrying an unconfirmed delivery.

Try a controlled model change

After setting both secrets above, deliberately request GPT-4.1 mini while keeping GPT-4.1 as the reference. These are fresh calls to a different model, not a natural provider incident. The real test determines the outcome; a mismatch is not guaranteed.

terminal · controlled comparison

npx --yes --package https://martingale.io/assets/cli/martingale-check-0.2.2.tgz martingale-check --model gpt-4.1-mini --reference gpt-4.1 --slack-webhook-env MARTINGALE_SLACK_WEBHOOK --slack-on all

Use your own gateway or endpoint

Set the key for your endpoint in step 1, then replace the example URL, model ID and reference below. This uses OpenAI-compatible Chat Completions. For Anthropic Messages, add --api anthropic and choose a matching Claude reference.

terminal · replace placeholders

npx --yes --package https://martingale.io/assets/cli/martingale-check-0.2.2.tgz martingale-check --base-url https://your-endpoint.example/v1 --key-env OPENAI_API_KEY --model your-model-id --reference gpt-4.1

Add --slack-webhook-env MARTINGALE_SLACK_WEBHOOK for mismatch notifications on a direct check. Slack notifications are not supported in routed mode.

Capabilities

Monitoring and alerts.

Canary suites

Scheduled hidden prompts with deterministic checks, run through your own provider account under a daily budget you set. Single calls and multi-step tool-using tasks, so a workflow that quietly starts taking more steps is caught too.

Drift detection

Statistical tests on canary results and live telemetry: pass rate, output length, latency, error rate, finish reasons, returned model, fixed-prompt token count, and Raven response fingerprints.

Passthrough proxy

Forwards requests to your model provider and records operational metadata, with streaming support.

Alerting

Slack and PagerDuty, deduplicated per endpoint and series, with severity thresholds and one resolution message when it clears.

Incident timeline

When it started, what changed, how confident we are, and which endpoints and tags are affected. Evidence is kept for the life of your account.

Dashboard + API

Endpoints, canaries, incidents, and settings in a dashboard and a REST API, so setup and investigation fit your workflow.

What Martingale will never do.

The guiding rule of the whole system: Martingale may lose its own telemetry, but it must never break, slow, or alter a customer's request.

Works with OpenAIAnthropicTogetherFireworksGroqOpenRouter

Route, block, or retry

The proxy never modifies, retries, or re-routes a request. The provider you chose is the provider that answers. The one exception is yours to turn on: the SDK can call the provider directly if the proxy itself is unreachable.

Wait on its own storage

Telemetry is buffered and flushed asynchronously. If the queue is down, events drop and your traffic continues.

Store your prompts

No prompt or completion content is stored unless you opt in. Canary text is Martingale-authored, so it can be.

Log your provider key

Your upstream key is never logged, printed, or persisted. Canary keys are stored encrypted.

Re-baseline silently

After a change persists for 7 days you can accept it as the new normal. The engine never does that for you.

Page you for noise

Multiple-comparison correction and persistence rules keep the alert channel worth watching.

Book a demo.

See Canary and Raven Alpha in the context of your product. Pick a time that suits you and tell us what you want to cover.

Book a demo contact@martingale.io

Is your endpoint serving the model you pay for?Check it now.