Reproducibility
Every eval response carries a system_fingerprint: an opaque token covering everything that determined the verdict — your supervision, our build, and the encoder underneath. It answers one question, has anything that determines this verdict moved?
It is a change detector, not an address. Reproduction is the model identifier’s job: the fingerprint reports that something moved, the identifier takes a verdict back.
§1What it is
An opaque token naming everything that determined your verdict: the learned state, plus the build and the encoder that read it. It changes when a fine-tuning round completes, when the shared baseline your model builds on is republished, and when we ship a change to the evaluation engine itself. It does not change between two requests against an unchanged model.
Treat the value as opaque. Compare it for equality against a fingerprint you saw earlier; do not parse it, and do not assume its shape is stable across model generations. It is built from several parts joined by punctuation, but which parts and in what order is a detail of composition, not a format to depend on.
§2What it tells you
Under an unchanged fingerprint, evaluation is deterministic: the same request returns a byte-identical response. The token covers every input to a verdict except the request itself.
§3What it does not tell you
§3.1It is not a reproduction key
Recording a fingerprint does not let you ask for that state again later — there is no request field that takes one. The field that does round-trip is model: it names the exact snapshot that answered, and sending it back reproduces the verdict. If you are storing one value for reproducibility, store that one — see §4.
§3.2It moves for reasons that are not about you
Most fingerprint changes are your own corrections landing. Some are not: a republished shared baseline, or a correction we ship to the engine. Those reach a pinned caller too, because a pin holds the model’s knowledge, not our software — see model identifiers §5. So a fingerprint that moves while your model identifier did not is us, not you.
§3.3A change does not imply a behaviour change
Every completed fine-tuning round mints a new fingerprint, including rounds that absorb supervision the model already agreed with. A changed token means re-check, not something broke. If your baseline still passes, nothing is wrong.
§3.4It is not comparable across accounts
Fingerprints are scoped to the account that owns the corrections. Two accounts’ tokens are not meaningfully comparable, and neither are tokens from different environments.
§4How to get reproduction
Use the model field. Every response names the exact snapshot that answered — even when you asked for the live a8 — and sending that value back as your model replays the verdict. A pinned snapshot is composed against the exact shared baseline it was trained over, so later improvements move the alias and leave your pin where it was.
So a stored result carries its own provenance without you tracking anything separately: keep the response’s model beside the verdict. The fingerprint is what tells you the live alias has moved on since; the model is what takes you back. See horizons.
§5Practical patterns
| You want to | Use |
|---|---|
| Cache verdicts and know when to invalidate | Key the cache on system_fingerprint. A new token empties it. |
| Detect that your model moved between two runs | Compare the fingerprints from the two runs. |
| Publish a number someone can re-derive | Record the response’s model identifier alongside the number. |
| Run a regression suite against a fixed model | Pin a horizon. Move the pin deliberately, on your schedule. |
| Confirm your corrections took effect | Watch the fingerprint change, or watch for the new snapshot in GET /models. |
Table 1. The fingerprint is a signal; the pin is a control. Most integrations want both.
model identifier is what takes you back to where it was.§6Related concepts
- Horizons — the horizon, and how pinning works.
- Fine-tuning — the usual reason a fingerprint changes.
- Reference — where the token sits in the response.