Distribute releases through your own database

In the fleet pattern your store carries the release and your runtimes apply it. One puller job holds the Agent key: it pulls the release promoted to an environment, verifies the whole chain, seals the result to a distribution key you generated, and writes one row per generation into your database. Every runtime holds only the distribution private key: it boots from the newest row, serves from memory, and hands a newer row to applyBundle(). No application host ever holds an Agent key or opens a connection to AirPrompter.

Before you build a fleet

You need:

• an Agent key for the environment, available only to the puller job as AIRPROMPTER_AGENT_KEY;
• a distribution key pair: airprompter keygen --purpose distribution writes the private half 0600 and refuses to write it inside a git checkout; register the public half on the environment’s Settings;
• the environment’s root document URL (--root-url in the CLI, fetchRoot here): a pinned key alone names the root, not the signing keys under it, so a puller without it refuses every manifest with unknown_signing_key;
• a table your puller can insert into and your runtimes can read; and
• a way for a runtime to learn that a row changed: a poll, a Postgres LISTEN, a message on your bus. The SDK does not choose this for you.

Set the fleet up

1. Generate the distribution key pair on a workstation and register the public half on the environment.
2. Create the release table in your database.
3. Deploy the puller with the Agent key and the distribution public key; confirm a row appears for the promoted generation.
4. Deploy each runtime with the distribution private key and no Agent key; confirm it boots from the newest row.
5. Promote a new version from the board and watch every runtime log activated (or staged under unlock-required).

The release table

The reference application keeps the table in SQLite; the shape is the point, not the engine. The same columns work in Postgres or as a DynamoDB item keyed on agent, environment, and generation.

CREATE TABLE IF NOT EXISTS agent_releases (
  agent_id       TEXT    NOT NULL,
  target         TEXT    NOT NULL,
  generation     INTEGER NOT NULL,
  release_digest TEXT    NOT NULL,
  bundle         TEXT    NOT NULL,   -- the sealed .apbundle as JSON text; ciphertext
  pulled_at      TEXT    NOT NULL,
  PRIMARY KEY (agent_id, target, generation)
);

What is in a row: the agent, the environment, the generation, the release digest, and the bundle: the signed manifest, every payload the manifest references, and the root document, sealed to the distribution public key. Nothing in the row is readable without the private half, so treat the column as private as the key. pulled_at is the puller’s own clock; the bundle carries the release’s own createdAt and notAfter.

The puller

import { SyncClient, nextPullDelayMs, pullBundle, trustedRootFromPinnedKey } from "@airprompter/agent-sdk";

const scope = { organizationId: env.AIRPROMPTER_ORG_ID, agentId: env.AIRPROMPTER_AGENT_ID, target: "prod" as const };
const client = new SyncClient({ baseUrl: "https://api.airprompter.com", agentId: scope.agentId, target: scope.target, apiKey: env.AIRPROMPTER_AGENT_KEY });
let trustedRoot = trustedRootFromPinnedKey({ purpose: "platform", environment: "prod", pinnedRoot: PINNED_ROOT_JWK });
const fetchRoot = async () => { const r = await fetch(env.AIRPROMPTER_ROOT_URL); return r.status === 200 ? r.json() : null; };
const distributionPublicKey = readDistributionPublicKey();   // 32 raw bytes; the puller never reads the private file

let edge = null;            // the puller's memory between pulls: pointer URL, two ETags, when the origin last answered
let unchangedStreak = 0;

async function pass({ skipPointer = false } = {}) {
  const held = table.newest()?.generation ?? 0;              // the floor: an older answer is reported, never stored
  const result = await pullBundle({ client, scope, trustedRoot, fetchRoot, minimumGeneration: held, now: () => new Date().toISOString(), distributionPublicKey, edge, skipPointer });

  if (result.status === "unchanged") {
    edge = result.edge;
    unchangedStreak += 1;
    return nextPullDelayMs({ outcome: result.status, unchangedStreak, intervalMs: 10_000, capMs: 300_000 });
  }
  unchangedStreak = 0;
  if (result.status !== "ok") {
    log(`pull: ${result.status}${result.reason ? ` (${result.reason})` : ""}`);   // nothing_promoted | refused | unavailable
    return 10_000;
  }
  trustedRoot = result.trustedRoot;                            // a rotated root is accepted only through the chain
  table.insert({ generation: result.generation, releaseDigest: result.releaseDigest, bundle: JSON.stringify(result.bundle), pulledAt: result.createdAt });
  edge = result.edge;                                          // commit the edge state WITH the row, never before it
  return 10_000;
}

process.on("SIGUSR1", () => void pass({ skipPointer: true })); // "check now": an operator, or a nudge from a channel you own

pullBundle returns one of five statuses: ok with the bundle, unchanged with via: "pointer" | "origin", nothing_promoted, refused with a reason (including generation_rollback when the server answered below minimumGeneration), and unavailable (unauthorized, forbidden, network, or an HTTP status). The reference puller also refuses to overwrite a stored row whose digest differs for the same generation, since that never happens on an honest server, and logs it loudly.

In Python the same call is pull_bundle(client=, scope=, trusted_root=, fetch_root=, minimum_generation=, now=, distribution_public_key=, edge=, skip_pointer=) with next_pull_delay_ms(...) beside it.

The pointer-first pull

After every promotion or rollback, AirPrompter writes a few hundred bytes per environment behind a CDN. The first pull reaches the API once and learns that pointer’s URL; from then on the puller hands the last result’s edge back and an idle interval is one CDN request answered 304, with no API call at all.

Pull	What happens	Cost
First	The API, once; learns the pointer URL and the manifest ETag	1 API request
Idle	Pointer with ETag: 304, or a generation the table holds, returned as unchanged via pointer	1 CDN 304
Moved	The pointer names a higher generation; the API is read conditionally; a new bundle	1 CDN read, 1 API request, the payloads
Nudged (skipPointer: true)	The API, conditionally: unchanged via origin or a new bundle	1 API request

nextPullDelayMs doubles the interval while nothing changes, capped at five minutes by default, and snaps back to the base interval on any change, refusal, or outage. Three rules keep the unsigned pointer honest: a stuck pointer has a bound (maxPointerAgeMs, one hour by default: past it the puller reads the API once, conditionally, so a pinned or unwritten pointer can hide a promotion for at most that long); nothing advances past an answer the API did not confirm (the pointer’s ETag is kept only when the pull ends in unchanged or ok); and a CDN outage or a malformed pointer falls through to the API.

Persist edge with the row it came back beside, in the same transaction, never before it: a saved manifest ETag for a row that was never written makes the next read a 304 and the row is never written.

The runtime

import { AirPrompterAgent } from "@airprompter/agent-sdk";

const { distributionKey } = readDistributionPrivateKey();   // { privateKey, publicRaw }; this file is 0600 and only here
const newest = table.newest();

const ap = await AirPrompterAgent.start({
  organizationId: env.AIRPROMPTER_ORG_ID,
  agentId: env.AIRPROMPTER_AGENT_ID,
  target: "prod",
  stateDir: "/var/lib/acme-support-bot",                     // this host's own encrypted store
  root: { pinned: PINNED_ROOT_JWK, hostedEnvironment: "prod" },
  distributionKey,                                            // opens every sealed bundle this host is handed
  ...(newest ? { vendoredBundle: { bundle: JSON.parse(newest.bundle) } } : {}),
  // no apiKey: this process never calls home
});

ap.onChange((change) => log(`generation ${change.generation} is live`));

let attempted = ap.generation;
let applying = false;
setInterval(async () => {                                    // or LISTEN/NOTIFY, or your bus: all end in applyBundle
  if (applying) return;
  const row = table.newest();
  if (!row || row.generation <= attempted) return;
  applying = true;
  try {
    const outcome = await ap.applyBundle(row.bundle);        // never throws on a bad bundle
    attempted = row.generation;                              // a refused or held-back row is not retried; only a newer one is tried
    log(`applyBundle(${row.generation}) → ${outcome.outcome}${outcome.reason ? ` (${outcome.reason})` : ""}`);
  } finally {
    applying = false;
  }
}, 3_000);

applyBundle() runs the same chain a network sync runs before a byte is staged and returns one of:

Outcome	Meaning
activated	Verified, staged, and live; generation is the new one
staged	Verified and staged; the environment’s policy is unlock-required, so it waits for your unlock, window, or hook
unchanged	The row is the generation this host already holds
held_back	The row is newer than what runs but below a generation this host rolled back from; heldBackBelow names it
refused	reason says why: a signature or hash failure, generation_rollback, expired, model_unavailable, daemon_attached, no_store

In Python: distribution_key= and vendored_bundle= at start, then ap.apply_bundle(row) returning the same outcome as a dict.

Expected result: stop the puller, start a second runtime on another port, and both serve the same generation from the table; promote from the board, and within one pull interval plus one watch interval every runtime logs activated. Under unlock-required they log staged and wait.

Variables your app fills from its own systems (start({ variables }), described on the TypeScript and Python pages) are registered in the runtime process and do not care where the release came from: a runtime that applies releases from your table fills them exactly as one that syncs from AirPrompter. The puller never renders and needs none. Fleet runtimes send no heartbeat, so the console cannot count them in the update drawer’s “filled on X of Y” line — check ap.status().variables and ap.prompt(tag).needs(...) when a runtime boots instead.

Restart and rollback

These are the day-to-day events a fleet meets: a host reboots, a bad release goes out, someone rolls back on the board.

• Restart. A runtime serves the applied generation from its own store before it looks at the table or the network. With the table unreachable it keeps serving; with the table empty and no store, start() refuses, as it should.
• A newer row at boot is an update like any other: verified, staged, and the apply policy decides. The held generation changes nothing. An older row at boot is refused and logged vendored_bundle_refused with the reason generation_rollback.
• Rolling back is await ap.rollback() on the runtime (or the rollback operation on a host daemon’s socket): it flips to the other slot instantly and is stamped as a forced downgrade. Handing an older row to applyBundle() is not a rollback; it is refused. After a local rollback, the generation you left is held back until AirPrompter moves past it, so a puller that keeps writing the same row does not re-apply it.
• A rollback in the console is a new, higher generation with the previous content. It lands on every runtime the same way a promotion does.

Rotate the distribution key

At least every 90 days (the bundle’s default lease): generate a new pair, register the new public key on the environment, let the puller write a fresh row sealed to it, and retire the old private key once every runtime has applied that row. Nothing already sealed is re-sealed by the console.

Learn about changes faster: the nudge (proposed)

The pointer-first pull is shipped. A nudge is proposed: on every promotion, rollback, dial move, or disable, AirPrompter would publish one content-free message (agentId, target, generation, issuedAt) to a channel you own, such as an Amazon SQS queue in your account, and your puller would run one pull with skipPointer: true. Nothing else changes: the pull verifies as always, the pointer poll keeps running, and a lost nudge costs one poll interval of latency. The reference puller already wires the same “check now” to SIGUSR1. The design note, its channel options, and its open decisions are in the SDK repository at docs/change-notification.md; the channel is not shipped yet and the note says so.

Troubleshooting

Every pull is refused with unknown_signing_key

The puller has a pinned key but no root document. Pass fetchRoot (the environment’s root document URL) or a full root document as trustedRoot.

The runtime logs held_back

This host rolled back below that row’s generation. Promote a newer generation in the console; the next row applies.

The runtime logs refused with model_unavailable

The release marks a slot’s model as required and this process did not declare it in models. The host keeps serving what it has; declare the model or promote a release that pins one you can call.

A promotion reaches runtimes only after several minutes

The puller’s interval stretched while nothing changed. Send SIGUSR1 (or your own nudge) to check now, lower capMs, or wait for the base interval to snap back on the next change.

Completion checklist

• Only the puller job holds AIRPROMPTER_AGENT_KEY; runtimes hold only the distribution private key.
• The public half of the distribution key is registered on the environment.
• edge is persisted in the same transaction as the row.
• A restarted runtime serves before it touches the table.
• An older row is refused with generation_rollback and your logs show it.

Next steps

Runtimes in a fleet still write the content-free spool; read how it leaves an estate with no egress in Telemetry. For hosts you operate by hand, the same pull, verify, and apply steps exist as CLI commands.