Understand and route telemetry

The runtime never sends a prompt, an answer, or an end-user identifier anywhere. What leaves a host is one row per minute per dimension set: counts, token totals, a 16-bucket latency histogram, error classes, output-check counts, and declared feedback outcomes. The rows are written to a local spool first and shipped later, so nothing about telemetry is ever on your request path. The spool is a published contract: any process, in any language, can write it and be picked up.

What a window row is

The spool lives at <stateDir>/airprompter/<agentId>/<target>/spool/telemetry/ (directory 0700, files 0600) as append-only newline-delimited JSON segments named seg-<instanceId>-<epochMinute>-<n>.ndjson. A segment rotates at the minute or at 1 MiB. One window row summarises one minute for one combination of prompt, version, arm, model, status, and error class:

{"type":"window","v":1,
 "minute":"2026-09-11T14:03:00Z",
 "instanceId":"i-7f3a…","instanceClass":"resident",
 "tag":"support.triage","versionId":"pv_…","arm":"candidate",
 "model":"claude-haiku-4-5","status":"ok","errorClass":null,
 "usageSource":"reported",
 "count":412,
 "latencyMs":{"buckets":[0,0,3,41,188,150,27,3,0,0,0,0,0,0,0,0],"sum":617384},
 "tokens":{"input":493200,"cachedInput":329600,"output":103000},
 "checks":{"passed":405,"failed":7},
 "outcomes":{"accepted":{"n":388,"sum":362},"rating":{"n":41,"sum":171}},
 "sdk":"agent-sdk-ts/0.2.9"}

Two other row types exist: refusal (a render refused, with the reason and generation) and dropped (segments evicted by the disk budget, with counts and bytes). Every row is validated against the protocol’s schema on the writer and again at ingest.

Before you decide how to ship it

You need to know three things about each host:

• whether it can open outbound HTTPS to AirPrompter, or only through a proxy, or not at all;
• whether several runtime processes share the host (each is its own instance and its own writer into one spool); and
• whether your platform team already runs an OpenTelemetry collector that should receive these counters instead.

Choose the uploader

Day-to-day, the choice is per host and rarely changes; pick it once when the host is provisioned.

Host	Uploader	Key on the host
One resident process with an Agent key	The runtime’s own in-process uploader (the default)	The Agent key it already has
Several processes on one host	airprompterd uploads every writer’s segments; runtimes attach with sync.mode: "daemon"	Only the daemon holds the key
A serverless function	The invocation’s rows are posted as one segment before the handler returns	The Agent key
A host with an OpenTelemetry collector	The OTel bridge exports windows to the collector, never to AirPrompter	None needed for telemetry
A host with no route out	airprompter export-telemetry packs the spool into a file; import-telemetry uploads it from a connected host	The key lives on the connected host

The in-process uploader

A resident host with no daemon runs the same uploader the daemon runs, in-process, on a timer with a random phase, under the runtime’s own short-lived grant. A failed pass backs off with full jitter and the next retries; acknowledged segments are deleted. ap.status().upload reports the sink, the last upload, the backoff, and sentSegments; await ap.uploadNow() runs one pass by hand, which the reference application does at the end of each run:

ap.spool.closeWindows(Date.now());                  // close the current minute so it can ship
console.log(await ap.uploadNow());                  // { uploaded, quarantined, dropped, held } or null when a daemon owns the spool

telemetry: { upload: false } (TelemetryOptions(upload=False) in Python) leaves the spool for a daemon or an operator’s export; the budget still holds and losses are still counted.

The grant a host holds is deliberately small: a presigned upload of at most 15 minutes and 1 MiB per object, bound to one prefix for this organization, agent, environment, and instance, accepting newline-delimited JSON only. It cannot read, list, or delete, and a row whose instance does not match the prefix is quarantined at ingest.

The host daemon

airprompterd runs one sync loop and one shared store per host and uploads every closed segment under the store’s spool, its own, the attached processes’, and any third-party writer’s, checked line by line first (a failing segment is quarantined whole, never sent). Passes run every --upload-interval-seconds (300 by default); --no-upload leaves the spool on disk; the upload operation on its socket runs a pass now. See The CLI and the daemon.

Send the counters to OpenTelemetry instead

Window fields follow the OpenTelemetry GenAI semantic conventions where one exists, so exporting is a renaming, not a redesign: model becomes gen_ai.request.model, the latency buckets become the gen_ai.client.operation.duration histogram, tokens become the airprompter.tokens sum by gen_ai.token.type, and the prompt, version, and arm become airprompter.prompt.tag, .version, and .arm. Every sum is delta temporality.

import { AirPrompterAgent } from "@airprompter/agent-sdk";
import { otlpUploadSink } from "@airprompter/otel-bridge";

const ap = await AirPrompterAgent.start({
  ...startOptions,
  telemetry: {
    uploadSink: otlpUploadSink({
      endpoint: "http://localhost:4318/v1/metrics",
      headers: { authorization: process.env.OTEL_TOKEN! },
      resource: { "service.name": "support-bot" },
    }),
  },
});

from airprompter_agent import AirPrompterAgent, TelemetryOptions
from airprompter_agent_telemetry.otel import OtlpUploadSink

ap = AirPrompterAgent.start(
    **start_options,
    telemetry=TelemetryOptions(upload_sink=OtlpUploadSink(endpoint="http://localhost:4318/v1/metrics", resource={"service.name": "support-bot"})),
)

# The daemon, for every writer on the host; no Agent key is needed on this path:
airprompter daemon --org org_… --agent agt_… --environment prod --root /etc/airprompter/root.jwk.json \
  --upload-sink otlp --otlp-endpoint http://localhost:4318/v1/metrics \
  --otlp-header 'authorization=$OTEL_TOKEN' --otlp-resource service.name=support-bot

The daemon reads --otlp-header 'name=$VAR' from its environment, never from the command line, so a collector token does not appear in a process list. The built-in exporter is OTLP/HTTP with JSON encoding; pass exporter for protobuf or gRPC. A collector that answers 4xx or 5xx has decided: the segment is dropped and counted (segment_dropped_by_sink). A collector that never answered has not: the segment is kept under backoff and the budget.

The disk budget is an invariant

What the spool can hold on a host is bounded, and the bound is published:

tree ≤ budget + (writers × 1 MiB open) + quarantine cap + exported cap

The budget (100 MiB by default; telemetry.spoolBudgetBytes, --spool-budget-bytes on the daemon) is closed, unsent segments, the only thing that grows with traffic when the registry is away. Each live writer holds at most one open segment of at most 1 MiB. quarantine/ and exported/ are capped at 10 MiB each, oldest first. Past the budget the oldest unsent segments are dropped and the loss is written as a dropped row and reported on the heartbeat, never silently. A full disk or an I/O error never reaches your request: the writer counts what it could not keep and says so in one dropped row the moment a write works again.

Run the proof on your own machine, with no registry behind it:

airprompter telemetry verify --budget 104857600 --sink-absent

It fills an in-memory filesystem past the budget with two writers, a crashed writer, a live writer, and an overfilled quarantine, prints the tree before and after, the eviction, the dropped row, and the bound, and exits 0 when the invariant holds.

Run inside a VPC with no egress

Every connection is opened by your side, outbound. Where that is not possible from an application host, move the connection somewhere it is:

1. Releases in through an egress host. Run the fleet pattern’s puller, or airprompterd, on a host that can reach AirPrompter, and let application hosts read rows from your database or attach to the daemon’s shared state directory. Application hosts then need no route out for releases at all. See the fleet pattern.
2. Counters out through your collector. Point the OTel bridge at a collector inside the VPC; the counters never leave for AirPrompter, and no grant is requested.
3. Counters out by file. On a host that never calls home, airprompter export-telemetry packs closed segments into one file and moves them to exported/; carry it to a connected host and airprompter import-telemetry uploads each segment under its own name, so importing the same file twice counts nothing twice.
4. Counters out through a proxy. Both SDKs take an injectable HTTP layer. In TypeScript pass fetch at start (also accepted by SyncClient and otlpUploadSink); in Python pass transport= (an httpx.BaseTransport, for example httpx.HTTPTransport(proxy="http://proxy.internal:3128")) to AirPrompterAgent.start, SyncClient, or OtlpUploadSink.

const ap = await AirPrompterAgent.start({
  ...startOptions,
  fetch: (input, init) => fetch(input, { ...init, dispatcher: proxyAgent }),   // undici's ProxyAgent, or any fetch-shaped function
});

Expected result: an application host with no route to AirPrompter still serves renders, still writes the spool, and its counters still reach the Metrics tab within a few minutes of the connected side’s next pass, whether by collector, file, or proxy.

Write the spool without our SDK

Teams that instrument the provider SDKs themselves write window rows straight into the spool directory; the daemon, or a runtime on the host, uploads them. The repository’s examples/spool-writer/ has a dependency-free TypeScript and Python writer that pass the same conformance vectors, and airprompter telemetry validate <segment> checks a segment line by line before it is ever quarantined, naming the line and the field, never a value.

Troubleshooting

The Metrics tab stays empty

Counters arrive a few minutes after upload. Check ap.status().upload.lastUploadAt and backoffUntil; a refused grant is the throttle and is logged. On a daemon host, airprompter status shows the uploader.

dropped rows appear in the spool

The host was away from the registry longer than the budget allows at its traffic. Raise spoolBudgetBytes, shorten the outage, or route through a collector.

Segments land in quarantine/

A writer produced a row the contract refuses. Run airprompter telemetry validate on the segment; the reason names the line and field.

The uploader is held

The grant issuer asked for a pause with a retry-after; the uploader honours it exactly and resumes.

Completion checklist

• You can say which uploader each host uses and where its key is.
• No row you inspected carries prompt text, output, or an end-user identifier.
• The budget and the dropped row are visible in your monitoring.
• Hosts without egress have a route for counters: collector, file, or proxy.
• Third-party writers validate their segments before deployment.

Next steps

Operate the daemon and the offline path with the CLI, or go back to the resident integration in TypeScript or Python.