draft-imran-systems-and-arguments-45Internet-Draft
← draft-imran-systems-and-arguments-45
Off the Bit Implementation 3.5 Category: System State: production Share: https://mosthofaimran.com/l/3-5
Air-gapped delivery pipeline

Air-gapped delivery pipeline

Shipping software into estates with no network path back, as one complete signed artifact, installed by an operator who cannot ask you a question.

Stack

OCI bundles · cosign · Helm · offline registry

Text

How to read this note. This is the reference design for delivering software into environments with no network path back to the vendor: the constraint, the decisions that follow, and the failure modes this kind of delivery has, with the standard answers to each. It is a solution path for a system like the one built rather than a disclosure of that system's internals. Site identities, counts, deployment topology and install history are deliberately absent.

The figures this page carried until 2026-09-03 (six sites, zero failed installs since 2024-09) were the handoff prototype's, as was its summary, and both are removed under erratum 7.20. No replacements are invented.

1. The constraint

There is no network path, and there is never going to be one.

Not a firewall with an exception process. Not a proxy someone can whitelist a host on. A disconnected estate, in a building you will not enter, running on hardware you will not see, installed by an operator who has never met you and may not share a language with you. You cannot ship a hotfix, read a log, attach a debugger, or ask what the screen says.

Everything difficult about this follows from one property: the feedback loop is weeks long and runs through a person who is not an engineer. In a connected system a bad release is found in minutes and fixed in an hour. Here it is found when somebody writes an email, and the email omits the part you need.

So the artifact has to be complete, it has to be verifiable by someone with no way to ask a question, and it has to fail in ways that are legible to a person reading an error message for the first time.

2. The decisions, and where each one is enforced

CONNECTED SIDE, WHERE EVERYTHING MUST ALREADY BE DECIDED build with egress denied every dependency, chart, migration, and the registry itself, in one artifact sign reproducible bytes THE GAP physical media DISCONNECTED verify, then preflight key arrived out of band install refuses if either fails install, idempotent safe to run twice support bundle the only diagnostic there is carried back by hand, weeks later, paraphrased Anything resolved at install time is a network call, and there is no network. Everything else follows.
Figure 1. The arrow crossing the gap goes one way. Everything on the left has to be right before it does, because nothing on the right can be corrected afterwards.

2.1. One artifact, genuinely complete. Every dependency, every base layer, every chart and migration, and the registry to serve them from, inside a single bundle. Anything resolved at install time is a network call in disguise. This is enforced at build time by denying egress in the pipeline, so a package that would have been fetched on the far side fails here instead, where somebody can fix it in ten minutes.

2.2. The same image ships everywhere, and configuration alone decides. No separate air-gap build. A special build for the hardest environment is the build that gets the least testing, and it is the one that has to work with no way to intervene. Connected estates run the same bytes, which means the air-gapped path is exercised continuously by everybody else.

2.3. Verification is refusal, not a warning. The installer checks the signature and the hash against a key distributed out of band, and stops if either fails. Not a prompt, not a flag to continue anyway. An override exists in every system that has one, gets used under deadline pressure, and the property being protected is the only reason a regulated site accepted a vendor artifact at all.

2.4. Install is idempotent and resumable. The operator will run it twice. Something will time out halfway, the media will be re-inserted, a step will be repeated. An installer that is only correct on a clean first run is an installer that will corrupt a site, and no one will be watching when it does.

2.5. Diagnostics are produced locally and designed to be quoted. Preflight checks run before anything is written and fail with messages meant to be read aloud or pasted into an email, not interpreted. A support bundle the operator can generate on demand is the entire remote debugging story, so it is a feature with a spec rather than a log directory somebody tars up.

3. Why this decides which customers exist

An air-gapped delivery path decides which customers can buy at all, which is a larger claim than hardening usually gets to make.

Central banks, defence and government estates, and telecom operators under sovereignty rules do not have a procurement route for software that requires a connection home. The requirement arrives as a precondition, and a vendor without an answer is filtered out before the technical evaluation starts.

The second effect is on everyone else. A product that installs with no network dependency installs cleanly in a restricted enterprise environment too, where the customer’s security team has opinions about egress but not a physical gap. The work done for the strictest customer lowers the integration cost for every customer below them, which is unusual: hardening usually taxes the common case, and this is one of the times it subsidises it.

4. Figures

This note reports none. The four that would matter are the number of sites under management, install success rate on first attempt, the version spread across the estate at a given moment, and elapsed time from release to a site being updated.

They exist and are not published here, because site counts and identities are the customer’s information rather than mine. The figures this page did carry (six sites, zero failed installs since a given month) were the handoff prototype’s, and erratum 7.20 removes them.

5. What I would do differently

Deny egress in the build pipeline on day one, not after the first failure across the gap. Failure 5.1 is entirely preventable and it is normally discovered the expensive way, because a connected pipeline is quietly forgiving and the gap is not. This is one line of CI configuration and it should predate the first bundle.

Version the bundle format itself, separately from the software. The installer on a site is whatever version arrived last, and it has to read a bundle produced by a much newer build. Treating the bundle as a versioned interface with its own compatibility rules avoids the failure where a site cannot be upgraded because it cannot read the thing that would upgrade it.

Write the operator’s runbook before the installer. The error messages, the preflight output and the support bundle are the product for the person actually doing the work, and they get treated as documentation to be written afterwards. Failure 5.5 is open, and the part of it that is addressable is entirely a writing problem that engineers schedule last.

Known failure modes

5.1, fixed. A hidden network dependency. Something resolves at runtime rather than at build time: a base image layer, a certificate revocation check, a license callback, an NTP sync, a DNS lookup with no fallback. It passes every test in a connected pipeline because the network quietly answers, and it fails only on the far side of the gap, where nobody can diagnose it. The answer is to build and test with egress denied by default, so that a missing dependency fails in CI rather than in somebody's data centre a month later.
5.2, fixed. Reproducibility drift. Embedded build timestamps, non-deterministic archive ordering, an unpinned transitive dependency, and the bundle the operator hashes no longer matches the bundle that was signed. The install is fine and the verification step fails, which is worse than it sounds: the operator learns that the check is unreliable, and the next time it fails they will be tempted to skip it. A verification step people route around has negative value.
5.3, open. Trust distribution is the unsolved part. A signature is only worth the process by which the public key reached the site, and that process is out of band by definition: a key fingerprint read over a phone, printed in a contract, or carried by the same person carrying the media. Rotating that key across a hundred disconnected sites is genuinely hard, and every scheme trades key freshness against the number of humans who have to do something correctly. Nobody has a clean answer.
5.4, accepted. Sites run different versions because updates travel at the speed of a person. Migrations therefore have to tolerate arriving several versions late, and upgrade paths need testing across a matrix that grows with every release. The decision is to support a bounded window of versions rather than pretend the estate is uniform, to test that window explicitly, and to refuse an upgrade that falls outside it rather than attempt it and corrupt data.
5.5, open. There is no telemetry, so failures are invisible until a person describes them. You learn about a problem weeks later, paraphrased by an operator who is not an engineer, in a message that omits the part you need. Local diagnostics narrow it: a support bundle the operator can generate and export, and error messages written to be quoted rather than interpreted. It does not close the gap, because the feedback loop still runs through a human and still takes weeks.
ImranImplementation 3.5production