# Two-service build, approval, and deployment

This walkthrough assumes one small Linux host running systemd and Caddy, two Go
services, and a trusted Gitea build runner. The host has the same root-owned Tend
binary at `/usr/local/bin/tend`; it does not need Git or Go.

## Prepare the host once

1. Create `/etc/tend/services`, `/etc/tend/environment`, and
   `/var/lib/tend/incoming`. The incoming and environment directories are
   root-owned mode `0700`.
2. Install one schema-2 file per service, one per-service deployment-event log,
   and one root-owned mode-`0600`
   environment file per service. A singleton's shared environment file must
   not define its configured listen key; its installed unit owns the live
   address and Tend overrides only the transient candidate. Give each
   singleton an imported Caddy handler and a root-owned handler template with
   exactly one `{{UPSTREAM}}` marker; this is the bounded traffic handoff while
   its fixed-address unit restarts.
3. Install the receive policy, forced `authorized_keys` entry, and exact sudoers
   rule from `examples/server/` after replacing every placeholder. The sudoers
   fragment preserves only `SSH_ORIGINAL_COMMAND`; the root receiver requires
   its exact protocol value and refuses every other requested command.
4. Pin the server host key in a dedicated client file. Do not accept a new key
   interactively during deployment.
5. Run `sudo tend check-server` and inspect the allowlisted service names.

## Build and approve

Trusted CI checks a clean pushed commit, uses a pinned Go toolchain, packages
twice, and requires byte-identical archives. It publishes the archive,
`RELEASE.json`, `BUILDINFO.json`, SPDX SBOM, and SHA-256 evidence.

The maintainer reads the candidate report and copies the exact approved digest
into the deployment command. Tend refuses a digest that is merely inferred from
the local file or differs from the produced value.

```text
tend push --target tend-deploy@server.example \
  --known-hosts /secure/tend_known_hosts \
  --identity /secure/tend_deploy_ed25519 \
  --service example-site \
  --artifact /approved/example-site.tar.gz \
  --sha256 <produced> --approve-sha256 <reviewed> --activate
```

Repeat independently with `--service docs-site`. Both builds and transfers can
run concurrently. The host-wide lock serializes only activation because both
services share Caddy. Tend does not stop the other application.

## Failure and recovery exercises

- Change a candidate marker: activation must fail, preserve the active release,
  and record the failed attempt without calling the candidate active.
- Make a Caddy template invalid: validation must fail and restore prior bytes.
- Make the public marker fail after an initially successful request: the
  activation window must catch the transient routed failure and restore the
  former slot/pointer.
- For a singleton, verify repeated canonical-origin requests remain successful
  while Tend routes to the candidate, restarts the fixed-address unit, and
  returns traffic to it. Inject failure at both Caddy reloads and require the
  prior handler and pointer to be restored.
- Stop the previous blue/green slot during the activation window: Tend must
  restore the old Caddy handler instead of accepting reduced continuity.
- Run `tend rollback --activate` for one service and verify the other service's
  units, pointers, and public origin did not change.
- Interrupt a transfer: no release becomes active and the incomplete incoming
  file is removed when the receiver exits.

After the soak, prune per service. Active and previous releases remain protected.