# Distribute releases through your own database

Run one puller that holds the Agent key and writes sealed release rows, boot every runtime from the newest row with only a distribution key, and understand the pointer-first pull, restart, and rollback behaviour.

In the fleet pattern your store carries the release and your runtimes apply it. One **puller** job holds the Agent key: it pulls the release promoted to an environment, verifies the whole chain, seals the result to a distribution key you generated, and writes one row per generation into your database. Every **runtime** holds only the distribution private key: it boots from the newest row, serves from memory, and hands a newer row to `applyBundle()`. No application host ever holds an Agent key or opens a connection to AirPrompter.

**What a compromised store can and cannot do**

Rows are `.apbundle` ciphertext sealed to the fleet’s distribution key. A tampered row fails the manifest signature or a payload hash and is refused before a byte is staged; a relabelled row fails its binding to agent and environment; an older row is refused below the generation the runtime holds. What a compromised store _can_ do is withhold newer releases; the runtimes keep serving what they hold, and the Fleet tab’s generation spread shows it.

## Before you build a fleet

You need:

- an Agent key for the environment, available only to the puller job as `AIRPROMPTER_AGENT_KEY`;
- a distribution key pair: `airprompter keygen --purpose distribution` writes the private half `0600` and refuses to write it inside a git checkout; register the **public** half on the environment’s Settings;
- the environment’s root document URL (`--root-url` in the CLI, `fetchRoot` here): a pinned key alone names the root, not the signing keys under it, so a puller without it refuses every manifest with `unknown_signing_key`;
- a table your puller can insert into and your runtimes can read; and
- a way for a runtime to learn that a row changed: a poll, a Postgres `LISTEN`, a message on your bus. The SDK does not choose this for you.

## Set the fleet up

1. Generate the distribution key pair on a workstation and register the public half on the environment.
2. Create the release table in your database.
3. Deploy the puller with the Agent key and the distribution **public** key; confirm a row appears for the promoted generation.
4. Deploy each runtime with the distribution **private** key and no Agent key; confirm it boots from the newest row.
5. Promote a new version from the board and watch every runtime log `activated` (or `staged` under unlock-required).

## The release table

The reference application keeps the table in SQLite; the shape is the point, not the engine. The same columns work in Postgres or as a DynamoDB item keyed on agent, environment, and generation.

```sql
CREATE TABLE IF NOT EXISTS agent_releases (
  agent_id       TEXT    NOT NULL,
  target         TEXT    NOT NULL,
  generation     INTEGER NOT NULL,
  release_digest TEXT    NOT NULL,
  bundle         TEXT    NOT NULL,   -- the sealed .apbundle as JSON text; ciphertext
  pulled_at      TEXT    NOT NULL,
  PRIMARY KEY (agent_id, target, generation)
);
```

What is in a row: the agent, the environment, the generation, the release digest, and the bundle: the signed manifest, every payload the manifest references, and the root document, sealed to the distribution public key. Nothing in the row is readable without the private half, so treat the column as private as the key. `pulled_at` is the puller’s own clock; the bundle carries the release’s own `createdAt` and `notAfter`.

## The puller

```ts
import { SyncClient, nextPullDelayMs, pullBundle, trustedRootFromPinnedKey } from "@airprompter/agent-sdk";

const scope = { organizationId: env.AIRPROMPTER_ORG_ID, agentId: env.AIRPROMPTER_AGENT_ID, target: "prod" as const };
const client = new SyncClient({ baseUrl: "https://api.airprompter.com", agentId: scope.agentId, target: scope.target, apiKey: env.AIRPROMPTER_AGENT_KEY });
let trustedRoot = trustedRootFromPinnedKey({ purpose: "platform", environment: "prod", pinnedRoot: PINNED_ROOT_JWK });
const fetchRoot = async () => { const r = await fetch(env.AIRPROMPTER_ROOT_URL); return r.status === 200 ? r.json() : null; };
const distributionPublicKey = readDistributionPublicKey();   // 32 raw bytes; the puller never reads the private file

let edge = null;            // the puller's memory between pulls: pointer URL, two ETags, when the origin last answered
let unchangedStreak = 0;

async function pass({ skipPointer = false } = {}) {
  const held = table.newest()?.generation ?? 0;              // the floor: an older answer is reported, never stored
  const result = await pullBundle({ client, scope, trustedRoot, fetchRoot, minimumGeneration: held, now: () => new Date().toISOString(), distributionPublicKey, edge, skipPointer });

  if (result.status === "unchanged") {
    edge = result.edge;
    unchangedStreak += 1;
    return nextPullDelayMs({ outcome: result.status, unchangedStreak, intervalMs: 10_000, capMs: 300_000 });
  }
  unchangedStreak = 0;
  if (result.status !== "ok") {
    log(`pull: ${result.status}${result.reason ? ` (${result.reason})` : ""}`);   // nothing_promoted | refused | unavailable
    return 10_000;
  }
  trustedRoot = result.trustedRoot;                            // a rotated root is accepted only through the chain
  table.insert({ generation: result.generation, releaseDigest: result.releaseDigest, bundle: JSON.stringify(result.bundle), pulledAt: result.createdAt });
  edge = result.edge;                                          // commit the edge state WITH the row, never before it
  return 10_000;
}

process.on("SIGUSR1", () => void pass({ skipPointer: true })); // "check now": an operator, or a nudge from a channel you own
```

`pullBundle` returns one of five statuses: `ok` with the bundle, `unchanged` with `via: "pointer" | "origin"`, `nothing_promoted`, `refused` with a reason (including `generation_rollback` when the server answered below `minimumGeneration`), and `unavailable` (`unauthorized`, `forbidden`, `network`, or an HTTP status). The reference puller also refuses to overwrite a stored row whose digest differs for the same generation, since that never happens on an honest server, and logs it loudly.

In Python the same call is `pull_bundle(client=, scope=, trusted_root=, fetch_root=, minimum_generation=, now=, distribution_public_key=, edge=, skip_pointer=)` with `next_pull_delay_ms(...)` beside it.

### The pointer-first pull

After every promotion or rollback, AirPrompter writes a few hundred bytes per environment behind a CDN. The first pull reaches the API once and learns that pointer’s URL; from then on the puller hands the last result’s `edge` back and an idle interval is one CDN request answered `304`, with no API call at all.

| Pull                         | What happens                                                                                     | Cost                                    |
| ---------------------------- | ------------------------------------------------------------------------------------------------ | --------------------------------------- |
| First                        | The API, once; learns the pointer URL and the manifest ETag                                      | 1 API request                           |
| Idle                         | Pointer with ETag: `304`, or a generation the table holds, returned as `unchanged` via `pointer` | 1 CDN 304                               |
| Moved                        | The pointer names a higher generation; the API is read conditionally; a new bundle               | 1 CDN read, 1 API request, the payloads |
| Nudged (`skipPointer: true`) | The API, conditionally: `unchanged` via `origin` or a new bundle                                 | 1 API request                           |

`nextPullDelayMs` doubles the interval while nothing changes, capped at five minutes by default, and snaps back to the base interval on any change, refusal, or outage. Three rules keep the unsigned pointer honest: a stuck pointer has a bound (`maxPointerAgeMs`, one hour by default: past it the puller reads the API once, conditionally, so a pinned or unwritten pointer can hide a promotion for at most that long); nothing advances past an answer the API did not confirm (the pointer’s ETag is kept only when the pull ends in `unchanged` or `ok`); and a CDN outage or a malformed pointer falls through to the API.

Persist `edge` **with** the row it came back beside, in the same transaction, never before it: a saved manifest ETag for a row that was never written makes the next read a `304` and the row is never written.

## The runtime

```ts
import { AirPrompterAgent } from "@airprompter/agent-sdk";

const { distributionKey } = readDistributionPrivateKey();   // { privateKey, publicRaw }; this file is 0600 and only here
const newest = table.newest();

const ap = await AirPrompterAgent.start({
  organizationId: env.AIRPROMPTER_ORG_ID,
  agentId: env.AIRPROMPTER_AGENT_ID,
  target: "prod",
  stateDir: "/var/lib/acme-support-bot",                     // this host's own encrypted store
  root: { pinned: PINNED_ROOT_JWK, hostedEnvironment: "prod" },
  distributionKey,                                            // opens every sealed bundle this host is handed
  ...(newest ? { vendoredBundle: { bundle: JSON.parse(newest.bundle) } } : {}),
  // no apiKey: this process never calls home
});

ap.onChange((change) => log(`generation ${change.generation} is live`));

let attempted = ap.generation;
let applying = false;
setInterval(async () => {                                    // or LISTEN/NOTIFY, or your bus: all end in applyBundle
  if (applying) return;
  const row = table.newest();
  if (!row || row.generation <= attempted) return;
  applying = true;
  try {
    const outcome = await ap.applyBundle(row.bundle);        // never throws on a bad bundle
    attempted = row.generation;                              // a refused or held-back row is not retried; only a newer one is tried
    log(`applyBundle(${row.generation}) → ${outcome.outcome}${outcome.reason ? ` (${outcome.reason})` : ""}`);
  } finally {
    applying = false;
  }
}, 3_000);
```

`applyBundle()` runs the same chain a network sync runs before a byte is staged and returns one of:

| Outcome     | Meaning                                                                                                                              |
| ----------- | ------------------------------------------------------------------------------------------------------------------------------------ |
| `activated` | Verified, staged, and live; `generation` is the new one                                                                              |
| `staged`    | Verified and staged; the environment’s policy is unlock-required, so it waits for your unlock, window, or hook                       |
| `unchanged` | The row is the generation this host already holds                                                                                    |
| `held_back` | The row is newer than what runs but below a generation this host rolled back from; `heldBackBelow` names it                          |
| `refused`   | `reason` says why: a signature or hash failure, `generation_rollback`, `expired`, `model_unavailable`, `daemon_attached`, `no_store` |

In Python: `distribution_key=` and `vendored_bundle=` at start, then `ap.apply_bundle(row)` returning the same outcome as a dict.

**Expected result:** stop the puller, start a second runtime on another port, and both serve the same generation from the table; promote from the board, and within one pull interval plus one watch interval every runtime logs `activated`. Under unlock-required they log `staged` and wait.

Variables your app fills from its own systems (`start({ variables })`, described on the TypeScript and Python pages) are registered in the runtime process and do not care where the release came from: a runtime that applies releases from your table fills them exactly as one that syncs from AirPrompter. The puller never renders and needs none. Fleet runtimes send no heartbeat, so the console cannot count them in the update drawer’s “filled on X of Y” line — check `ap.status().variables` and `ap.prompt(tag).needs(...)` when a runtime boots instead.

## Restart and rollback

These are the day-to-day events a fleet meets: a host reboots, a bad release goes out, someone rolls back on the board.

- **Restart.** A runtime serves the applied generation from its own store before it looks at the table or the network. With the table unreachable it keeps serving; with the table empty and no store, `start()` refuses, as it should.
- **A newer row at boot** is an update like any other: verified, staged, and the apply policy decides. The held generation changes nothing. An older row at boot is refused and logged `vendored_bundle_refused` with the reason `generation_rollback`.
- **Rolling back** is `await ap.rollback()` on the runtime (or the `rollback` operation on a host daemon’s socket): it flips to the other slot instantly and is stamped as a forced downgrade. Handing an older row to `applyBundle()` is not a rollback; it is refused. After a local rollback, the generation you left is held back until AirPrompter moves past it, so a puller that keeps writing the same row does not re-apply it.
- **A rollback in the console** is a new, higher generation with the previous content. It lands on every runtime the same way a promotion does.

## Rotate the distribution key

At least every 90 days (the bundle’s default lease): generate a new pair, register the new public key on the environment, let the puller write a fresh row sealed to it, and retire the old private key once every runtime has applied that row. Nothing already sealed is re-sealed by the console.

## Learn about changes faster: the nudge (proposed)

The pointer-first pull is shipped. A **nudge** is proposed: on every promotion, rollback, dial move, or disable, AirPrompter would publish one content-free message (`agentId`, `target`, `generation`, `issuedAt`) to a channel you own, such as an Amazon SQS queue in your account, and your puller would run one pull with `skipPointer: true`. Nothing else changes: the pull verifies as always, the pointer poll keeps running, and a lost nudge costs one poll interval of latency. The reference puller already wires the same “check now” to `SIGUSR1`. The design note, its channel options, and its open decisions are in the SDK repository at `docs/change-notification.md`; the channel is not shipped yet and the note says so.

## Troubleshooting

### Every pull is `refused` with `unknown_signing_key`

The puller has a pinned key but no root document. Pass `fetchRoot` (the environment’s root document URL) or a full root document as `trustedRoot`.

### The runtime logs `held_back`

This host rolled back below that row’s generation. Promote a newer generation in the console; the next row applies.

### The runtime logs `refused` with `model_unavailable`

The release marks a slot’s model as required and this process did not declare it in `models`. The host keeps serving what it has; declare the model or promote a release that pins one you can call.

### A promotion reaches runtimes only after several minutes

The puller’s interval stretched while nothing changed. Send `SIGUSR1` (or your own nudge) to check now, lower `capMs`, or wait for the base interval to snap back on the next change.

## Completion checklist

- Only the puller job holds `AIRPROMPTER_AGENT_KEY`; runtimes hold only the distribution private key.
- The public half of the distribution key is registered on the environment.
- `edge` is persisted in the same transaction as the row.
- A restarted runtime serves before it touches the table.
- An older row is refused with `generation_rollback` and your logs show it.

## Next steps

Runtimes in a fleet still write the content-free spool; read how it leaves an estate with no egress in [Telemetry](https://airprompter.com/docs/integrations/telemetry). For hosts you operate by hand, the same pull, verify, and apply steps exist as [CLI commands](https://airprompter.com/docs/integrations/cli).

---

Canonical: https://airprompter.com/docs/integrations/fleet