Cutting an ISO Release

The whole pipeline lives in archiso/. CI (.github/workflows/build.yml) only compile-checks — it does not build or publish ISOs. Releases are cut by hand.

The steps

1. Pre-flight the tree. One script now answers what used to be five manual checks:

tools/preflight.sh          # working tree + index, against HEAD
tools/preflight.sh --staged # only what `git commit` would take
tools/preflight.sh --self-test   # prove the checks can still fail

It refuses a commit that cannot ship, and every finding is a fact about the repo rather than a style opinion:

Run --self-test if a finding surprises you: a matcher that matches nothing prints ok, so a broken check and a clean tree look identical.

Still worth an eye by hand: untracked source files, since git commit -a can push a tree that does not build.

git status --short --untracked=all -- synui synapd synguard synsh synnet synapse_kmod

2. Bump the version. iso_version in archiso/profiledef.sh is what the ISO's filename is built from — but the version is also written into the boot menu text in five more files, and nothing checks that they agree. Miss one and the image boots to a menu naming the previous release:

cd archiso
sed -i 's/0\.2\.9\.4/0.2.9.5/g' profiledef.sh grub/grub.cfg grub/loopback.cfg \
    syslinux/archiso_head.cfg syslinux/archiso_pxe-linux.cfg syslinux/archiso_sys-linux.cfg
grep -rn '0\.2\.9\.4' --include='*.sh' --include='*.cfg' . | grep -v '^./work/'

The grep should print nothing. (work/ is the previous build's tree and is regenerated.)

Do not touch SYNAPSEOS_VERSION in build.sh; see the version model. Bump the pkgrel of any component whose code changed.

3. Build (~20–30 min: compiles llama.cpp, downloads a ~4 GB GGUF, runs mkarchiso):

sudo ./archiso/build.sh --no-clean

The ISO is signed by default. Which key is named by archiso/release-key.fingerprint, so there is nothing to type and nothing to remember; the key's passphrase is prompted for near the end, so this is not unattended. build.sh proves the key is usable in its preflight, not at the end — a host without the key stops in the first ten seconds rather than after half an hour of building.

Use --no-sign for a throwaway image. And if an ISO ends up unsigned anyway, that never costs a rebuild — the signature is detached, so the image is already the right one:

./archiso/sign-iso.sh 0.3.0        # or no argument for the newest in out/

See Signing below for the key itself. publish-release.sh warns about an unsigned release rather than refusing.

Output: archiso/out/SynapseOS-<ver>-x86_64.iso.

The ISO ships a CPU llama build on purpose, and the default flags give you that. A CUDA or ROCm build links the shipped libggml against the build host's driver stack, so synapd exits 127 on any machine without it — including every VM. --gpu=vulkan is the exception: it links only the Vulkan loader and runs on any AMD/Intel card.

Regardless of the ISO's own backend, the build also packages synapse-llama-cuda into the repo if nvcc is on the host, and synapse-llama-vulkan if glslc is — so an installed machine can switch onto its GPU later. Those go in the repo, never into packages.x86_64.

4. Verify the right packages actually made it in. This is the step that catches a silently stale build:

grep '^synui ' archiso/work/iso/arch/pkglist.x86_64.txt   # → synui 0.1.0-<pkgrel>

If that pkgrel isn't the one you just bumped, the ISO is stale — do not ship it.

5. Publish. GitHub caps release assets at 2 GiB, so the ISO is split:

./archiso/publish-release.sh <ver>

This splits the image into 1900 MiB .part* files and runs gh release create v<ver>. Recent ISOs are ~6.5 GB → 4 parts.

It uploads <iso>.asc alongside them when the build was signed, and verifies that signature before publishing — a signature that does not check out stops the release rather than shipping. With no .asc it prints a loud warning and publishes anyway, because a test build is a legitimate thing to release.

6. Mirror the whole image. The split is GitHub's constraint alone. The download people are pointed at is a single file, and it lives on Cloudflare R2 with the Internet Archive as the mirror:

./archiso/publish-mirrors.sh <ver>

This uploads the ISO and its .sha256, .b2sum and .asc to both, then checks what each host actually serves — the size, and the hash of the last mebibyte by range request — before reporting either as published. An upload tool's exit status says the transfer returned, not that the object is servable: a bucket whose public access was never switched on, or a custom domain pointing somewhere else, exits 0 and serves nothing.

Re-running it is cheap. Anything already uploaded at the right size is skipped, so an interrupted 4.6 GB transfer is resumed rather than restarted, and --skip-r2 / --skip-ia publish to one mirror alone. The Archive's torrent is built by its derive queue some time after the upload, so it is announced as pending rather than printed as a finished link — check it exists before the download page links it.

Setup for both tools is under Mirrors in archiso/README.md.

7. Update the download page. publish-release.sh generates the release body with the right links on its own, but the site's download section carries the version in the URL and is edited by hand — it is the one place per release that is. The wiki points at the site rather than at a versioned file, so it does not need touching.

Signing

Releases from 0.2.9.5 carry a detached GPG signature. The checksum published in the same release proves the download is not corrupt and nothing more: whatever could alter the ISO could alter the checksum beside it. The signature is the part that says who built it, and the public key is published on soslinux.org rather than in the release, so the two reach people by different roads.

The signing key is a dedicated release key — not a personal identity, so it can be rotated or revoked without touching anything else, and a compromised build host costs the release key rather than your name.

gpg --quick-generate-key "SynapseOS Release Signing <releases@soslinux.org>" \
    ed25519 sign 3y
gpg --list-secret-keys --keyid-format=long          # the fingerprint
gpg --armor --export <fingerprint> > synapseos-release-key.asc

Give it a passphrase; keep the revocation certificate gpg writes to ~/.gnupg/openpgp-revocs.d/ somewhere that is not the build machine. The exported public half belongs in the website repo as public/synapseos-release-key.asc, with its fingerprint on the download page.

The fingerprint goes in archiso/release-key.fingerprint, which is committed: a fingerprint is public, and the file is what lets the key be named without anyone remembering to name it. SYNAPSE_SIGNING_KEY=<fpr> sudo -E ./archiso/build.sh still overrides it for a build that is not signing with the project key.

Signing lives in one script, archiso/sign-iso.sh — build.sh calls it, and you run it directly to sign an image built without a signature:

./archiso/sign-iso.sh              # newest ISO in archiso/out/
./archiso/sign-iso.sh 0.3.0        # by version
./archiso/sign-iso.sh --check-key  # prove the key is usable; sign nothing

It signs as the invoking user rather than as root — the build runs under sudo and root's keyring holds no release key — and names the key explicitly, because gpg --detach-sign with no -u signs with whatever secret key comes first. It verifies the signature it just wrote, against the file that will actually ship; publish-release.sh verifies again before uploading, and refuses an .asc that does not check out.

0.3.0 shipped unsigned, which is why signing is now the default rather than a flag: --sign and sudo -E and an environment variable were three things to remember at the end of a long build, and remembering is not a mechanism.

Reproducible builds

A build of one commit is stamped with that commit's time, not the clock: tools/source-date-epoch.sh gives SOURCE_DATE_EPOCH — the component's last commit for a package, HEAD for the image — and build.sh and build-all.sh both use it. Each package is built at the same path every time, because .BUILDINFO records the build directory.

Check the packages — builds each component twice from a clean clone of HEAD and compares them file by file:

tools/repro-check.sh --all --skip synapse-llama
tools/repro-check.sh synui synguard          # or just these

Check an image — build the same commit twice, keeping the first ISO aside, then:

tools/iso-repro-diff.sh first.iso second.iso

It prints every differing path under the cause that explains it, and anything unexplained on its own. Exit 0 means identical, 1 means only listed causes, 2 means something unexplained. Neither script needs root.

--no-clean is safe (it wasn't always)

Two stale-artifact bugs used to make builds silently "succeed" on old output. Both are fixed (3b6b747), but they're worth knowing because they're the shape of failure to expect here:

1. --no-clean used to preserve mkarchiso's work/ too, not just the expensive build/ (llama.cpp). mkarchiso's per-stage stamp files then made it skip straight to "Done!" in about two minutes — reusing the previous ISO and its old pacstrapped packages, including an unbumped synui. A false-positive exit 0. Now build.sh always wipes work/ and gates only build/ on --clean.

2. Git-sourced packages left a stale src/<pkg> that broke the next build. See the src collision hazard.

Two earlier ones (ec361d2, bf5d733): create_source_tarball omitted data/ so synui's install_data(data/wallpaper.png) failed; and build.sh chowned the ISO files but not the out/ directory (mkarchiso creates it root-owned), so publish-release.sh — which runs as your user — couldn't write the split parts.

The collector drift, and why it is now deduped

create_source_tarball existed in two places — build-all.sh and archiso/build.sh — and every single fix to it landed in one and not the other. Four times out of four. The symptom was always the same shape: a file missing from the tarball, a build that fails later and somewhere else, and a fix that "was already applied".

Both collectors now delegate to the component's own <pkg>/mktarball.sh when one exists (8629c32), and build.sh treats a failing mktarball.sh as fatal rather than falling through to a stale tarball. When you add a package that needs particular files, write it a mktarball.sh — do not teach the two collectors about it.

Testing without hardware

QEMU_RAM=8G ./archiso/build_scripts/qemu-test.sh   # auto-detects the newest ISO

Uses KVM when available, boots UEFI via OVMF (falling back to BIOS), and attaches a persistent 20 GB test disk. Give it 8 GB+ of RAM when the model is embedded. Kernel and boot output are mirrored to the serial console — View → serial0 in the QEMU window.

The default CPU build is already the right one for QEMU — and for release.

Release history

See also: Building and Packaging.