FactRelay Docs
FactRelay Measurement Standard
The rules every FactRelay number obeys — declared surfaces, frozen prompt sets, separated panels, repeated sampling, immutable snapshots, and n/N reporting with controls.
Generative answers are non-deterministic and platform-dependent. A single screenshot proves only that one answer occurred once. This standard defines how FactRelay produces numbers that can be compared over time and defended under scrutiny. It applies to paid engagements, self-audits, and publications alike.
1. Declared measurement surfaces
Every observation names its surface: the specific access path, provider, and conditions it was captured under.
| Surface class | Example | May be described as |
|---|---|---|
| Authorized API with web search | OpenAI Responses API web search | “OpenAI API web-search measurement” — never “your ChatGPT ranking” |
| Search API | Perplexity Search API | Structured search results — never consumer answers |
| First-party site data | Google Search Console (client-authorized) | The client’s own property data |
| Consumer UI sampling | ChatGPT / Gemini / Copilot / Claude apps | Manual, account-, region-, and time-stamped observations |
API and consumer interfaces are different surfaces: consumer products may rewrite queries and use location or memory. Results from different surfaces are never merged, and no cross-surface “AI score” exists in our reporting.
2. Frozen prompt sets
Prompts are versioned and frozen before baseline sampling. Natural-panel prompts are submitted byte-identical on every run — no appended numbering, no “please cite sources” riders. Changing a prompt creates a new version; versions are never silently mixed.
Three panels, never merged:
- Natural panel — the question as a buyer would ask it;
- Citation-diagnostic panel — explicitly requests sources; run separately because the request changes behavior;
- Controls — brand-free prompts measuring background drift.
3. Repeated sampling
Each prompt runs in 3–5+ independent sessions per window, distributed across adjacent time blocks. Results are reported as n/N per prompt per surface — “4/5 runs contained the misstatement” — with sample sizes always visible. No single-run conclusions, no pseudo-precise rankings.
4. Immutable snapshots
Every run archives: exact prompt and hash; prompt-set version and panel; provider, surface, and model identifier; country, language, timezone, timestamp; whether web search triggered; the full raw response and provider payload; citations with normalized URLs; retry lineage, schema version, and cost. Raw responses are never overwritten. When parsers improve, results are recomputed from snapshots — history is preserved.
5. Retests are like-for-like
A retest re-runs the same prompt-set version on the same surface under the same rules. Reports show baseline vs retest n/N beside the controls, list the published actions in between, and state interpretation limits. Direction is evidence; causation is not claimed. Null results are reported as results.
6. Prohibited representations
- Cross-platform composite scores or “overall AI rank”;
- API results presented as consumer-experience rankings;
- Single observations presented as stable facts;
- Effect claims without a control comparison;
- Any guarantee of future ranking, mention, or recommendation.
The public demo case shows the customer-facing comparison and recheck without publishing internal run records.