ProductOperate
Change verificationpreview
The same checks before and after a deploy, with the result on the pull request
A change's claims, measured before and after it rolls.
chaos verify takes what a change says must hold, in a form a machine can check, runs the same bench before and after the roll, and posts a verdict per claim on the pull request. The first time we ran it on our own release, it caught a problem. It runs on our own changes; on yours it starts as a pilot.
In early access. A 30-minute call is the way in; there is no SLA during early access.
A deploy goes out, the pipeline is green, and nobody measured whether the change did what it said. Regressions turn up later, in production, from users.
The verdict says what was measured and nothing more: a reader decides what it means.
How it works
- 01
Where it runs
- The bench runs against your deployment before and after the roll, and the verdict is posted on the pull request through the GitHub App. On our own releases today; on yours as a pilot.
- 02
What it reads
- The claims the change makes, the bench's checks and the monitors a claim names. A pilot on your system measures read-only, with no load and no faults on production.
- 03
What you approve
- You write the claims. The verdict says only what was measured; the person reviewing the pull request decides what it means.
Each line says how far it is built today. Live means anyone with an account uses it; preview means it runs in early access or as a pre-release; coming means it is written down and not built yet.
What you get
Claims a machine can check
A status, a latency bound, an error rate at a load, an unchanged API contract. A claim written only in words is listed and never counted as a pass.
previewThe same bench, before and after
Every check of the bench against the deployment, and the monitors a claim names, run before the roll and again after it.
previewA verdict per claim, on the pull request
Pass or fail for each claim and each dimension, posted where the change is reviewed, with the record behind it as an unsigned draft.
previewLoad, faults and planted regressions
Load and faults on a deployment built for the change, regressions planted on purpose to grade the verifier, and one verdict per change required of our agents. Not built yet.
coming
The first verification, on pull request #251 of our own platform, as it ran.
In the console
$ mise run verify:before 251pull request 251 at 26202e4cf865: 4 claimssurface-1 `http_healthz` passessurface-2 `grpc_protocol_health` passescontract-1 the public API is unchangedtext-1 chaos verify posts a verdict on this pull request after it rollsPASS http_healthz http 206.5 ms 200 OK {"status":"ok"}PASS grpc_protocol_health grpc 115.0 ms status 1PASS http_mcp_tools http 165.0 ms tools=21# 41 more checks, every one passed44 passed, 0 failed# 39 minutes later, after the change rolled$ mise run verify:after 251PASS http_healthz http 308.0 ms 200 OK {"status":"ok"}PASS grpc_protocol_health grpc 167.9 ms status 1FAIL http_mcp_tools http 8.3 ms an anonymous tools/list answered 401 … want 401 naming resource_metadataaddress shortened# 41 more checks, every one passed43 passed, 1 failed### Verdict: failverdict posted on pull request 251
The record
Pull request #251, head 89339c77f63f
What we expected
- pass
http_healthzpasses200 in 308 ms - pass
grpc_protocol_healthpassesserving, in 168 ms - passthe public API is unchanged98 operations compared, none changed
- not measuredthe verdict is posted on the pull requestplain-language claims are never measured
Checked by
- failnothing that passed before fails now44 checks passed before, 43 after:
http_mcp_toolsfailedWhy it failed
The
http_mcp_toolscheck had been updated by an earlier pull request that was merged but not yet live, so the verifier expected a sign-in answer the platform didn't give yet. The change we tested didn't break it. The verdict still says fail, because it records what was measured, and a person decides what that means. - not measuredno new findingno load or faults on a bench yet
- n/amust-fail checks still failthe change named none
This record is an unsigned draft. We only observed; we didn't plant a known bug to grade the verifier against. Planted bugs, signing and storage aren't built yet (RFC 0046, RFC 0047).
The first chaos verify run on our own platform, every line as it was recorded. The verdict is a fail, and an unsigned draft.
Every claim on this page has its evidence: the documents that define it, the docs that say how to use it, and the API it runs on.
Proof and links
Documents
API
- GitHubliveThe verdict is posted as a comment on the pull request.
- MCPcomingThe checks as tools an AI assistant can call are not built yet.
The evidence rule: a model saying it succeeded is never evidence. What counts is what was measured, by what, and when.
Your data, your call
On your system a pilot measures read-only: monitors from our cloud or your agent, no load and no faults on your production.
Logs record who did what and the outcome, never content. The platform's own models are self-hosted; a third-party model provider is one you connect with your own key, and it bills you directly, with no line or markup from InOrbit. Separately, our own DPAs with model providers are in progress, none signed yet. InOrbit is not certified: no auditor has attested its controls yet, and the known gaps are listed.
Questions, answered plainly
Why show a run that failed?
Because it is what was measured. A check had changed with an earlier pull request that was merged and not rolled; the verdict records the fail and says why.
Can I run it on my changes?
As a pilot: one change, agreed in writing, measured before and after on your system, read-only.
Is the verdict signed?
Not yet. Records are unsigned drafts, marked so, until signing and storage are built.
Next step
See it on your own system
Book 30 minutes with an engineer. We look at your system together and, where it fits, open early access for your team.