tend/docs/DOGFOOD_EVIDENCE.md

# Preview dogfood evidence

This is maintainer-run operational evidence, not an independent audit or a
general reliability claim. It preserves each dated campaign and its limitations
instead of rewriting earlier observations as though later fixes had already
existed.

## Operation-scoped lease compatibility campaign — August 18, 2026

The candidate-lease compatibility fix completed trusted verification, exact
candidate reproduction, live activation, rollback, reactivation, and a bounded
failure injection on the production Linux/systemd/Caddy host. Conventional
deployment state remained schema 1 throughout. Singleton operation identity
used the separate adjacent lease introduced by this candidate.

### Assessed Tend candidate

- Private implementation source commit:
  `789976ce766fd457ca10762540135e4e0e74cfe3`.
- Version: `v0.2.0-preview.2`.
- Linux/amd64 binary SHA-256:
  `d0109bf037e493c7a046defe37818c3c1811806360b99c9a0910213665bd95c5`.
- Release-candidate archive SHA-256:
  `2c225e0b9dd0be2d36fda62ab8d0b52cd9b28f9336c60dfb5edf3e715f450f95`.
- Toolchain: Go 1.26.6, `CGO_ENABLED=0`, `-trimpath`.
- Trusted verification run 366 and release-candidate run 369 passed for the
  exact commit. The release workflow built the archive twice with identical
  bytes and published checksums, build metadata, and an SPDX SBOM.
- The forge upload action reported a finalized 3.18 MB artifact, but both
  documented artifact-list endpoints returned an empty result. A clean
  independent clone therefore built the package twice and reproduced the
  workflow's exact archive digest before deployment. This is recorded as an
  artifact-publication limitation, not described as a successful consumer
  download.
- The previous installed Tend binary remained retained by checksum before the
  candidate replaced the active tool. Server policy validation then passed.

### Live maintenance and failure evidence

Gamertan activated archive
`c17b4db1a4e2fd5406b392ec72195b947272097e4d1bfcec9d700d0889c3a4c6`,
rolled back to its recorded previous archive, and reactivated the first archive.
Both blue/green units remained active with zero restarts. Reconciliation was
settled after every transition.

The Sandwich Hime website performed the same sequence with archive
`d418f93ced3307fa788f5d2209b5f09f16bceabec4a9698a9beec6a05e439f35`.
Every successful forward and rollback operation removed its operation-scoped
candidate lease and transient unit. A final intentional local-smoke failure
asked the otherwise valid candidate for an impossible marker. The candidate
failed before Caddy or release-pointer mutation, stopped, removed its lease,
recorded the failed attempt without changing the active release, and reconciled
as settled and consistent.

After the campaign, Gamertan, its news and case-study indexes, Sandwich Hime,
its documentation and tutorial, the Gamertan-mounted Sandwich route, and EQL
health all returned HTTP 200. Caddy, both Gamertan slots, and the installed
Sandwich Hime service were active with zero restarts and no warning-or-higher
journal entries during the campaign.

Observatory is not claimed by this candidate campaign. Its current dogfood
deployment is a Docker Compose singleton, which remains outside Tend's
versioned systemd/Caddy strategies. The earlier three-service campaign remains
valid for its assessed candidate, but the current commit must not be tagged
until Observatory either returns to a supported topology or a separately
reviewed Compose strategy completes the same activation, rollback, and failure
gates.

## Final Preview 2 code-candidate campaign — August 18, 2026

The final v0.2 Preview 2 implementation candidate completed two explicit
rollback-and-reactivation cycles for Gamertan, the Sandwich Hime website, and
Gamertan Observatory on the production Linux/systemd/Caddy host. The candidate
routed singleton traffic to the already-proven transient process while the
fixed-address installed unit restarted, then restored the canonical upstream
only after loopback and public-origin validation.

### Assessed Tend candidate

- Private implementation source commit:
  `1fd3b9904c46e817c244196dd5d5a90921ca81a2`.
- Version: `v0.2.0-preview.2`.
- Linux/amd64 binary SHA-256:
  `adb4753d4e865775d50d618c999f10dfac9a0de945ccfd9d0b8f2e80d65f4597`.
- Gitea release-candidate archive SHA-256:
  `2bdfee2168cb16883d524cf9703adfa6ed5bc2bf44fbbb896993acc5fe6969ec`.
- Toolchain: Go 1.26.6, `CGO_ENABLED=0`, `-trimpath`.
- Trusted verification run 286 and release-candidate run 287 passed. The
  archive's checksums, SPDX SBOM, embedded version, commit, clean VCS state,
  target, and Go build information were independently rechecked before host
  installation.

### Application maintenance artifacts

- Gamertan archive SHA-256:
  `4700b075640b8b2fb5c17e0e02cf8d96ee67ceee10fe76d108c8b411983a88aa`.
- Sandwich Hime website archive SHA-256:
  `46b4da41cf6703fb8818e7d25e3a9c13e57cf700f5608b1888c4adb3352e4d38`.
- Observatory preview 12 archive SHA-256:
  `9ef0ddd8ec25d8fb75d6a6887e3ba874df7ebba16d3254f6250fcc646f4fd7f4`.

Every service ended with the intended current release active and the older
release retained as the explicit rollback target. Gamertan returned to its
green slot; both singleton services returned to their fixed addresses.
Candidate ports and the shared lock were free afterward. Caddy and all five
installed application units were active with zero restarts and no failed
units.

### Continuity and deployment evidence

A seven-minute workstation probe sampled the Gamertan origin, both Sandwich
Hime origins, Observatory, and EQL health 1,606 times each throughout the
campaign. It observed no HTTP failure status. One simultaneous client-side
disconnect affected all five destinations during a reload. A controlled
90-second replay therefore observed the same validated no-content-change Caddy
reload from both the workstation and an independent Linux host. The Linux host
recorded 450 successful responses and zero failures for every origin; the
workstation alone repeated one common-mode URL transport error across every
destination. Caddy retained the same PID with zero restarts and continued
serving unrelated requests. The common-mode workstation event is recorded as
an observer-path limitation, not server downtime.

The authoritative deployment-event files finished at 18 Gamertan events, 14
Sandwich Hime events, and 16 Observatory events. Observatory's agent cursor for
each stream exactly equalled the corresponding file size, proving complete
consumption. The agent and applications reported no warning-or-higher journal
entries during the campaign. Representative public routes returned HTTP 200,
and the EQL origin remained healthy.

### Findings closed before this campaign

Two earlier pre-activation artifacts exposed a mode-restoration defect under a
hardened root umask. They failed before route or pointer mutation. Tend now
reapplies validated archive modes explicitly and tests extraction under umask
`0077`.

The older singleton strategy briefly exposed an unavailable fixed upstream and
produced transient Observatory-agent `502`s. The final candidate's routed
handoff removed that failure in repeated production activation and rollback.

This evidence update changes VCS build metadata but not deployment logic. The
release policy therefore still requires one last maintenance pass with the
exact evidence-bearing candidate before the signed public tag is created.

## Restricted multi-service campaign — August 16, 2026

Tend's v0.2 implementation candidate completed the same maintenance contract
through the restricted transport for two independently configured services on
one Linux/systemd/Caddy host.

### Assessed Tend implementation

- Implementation source commit:
  `840b77da708bbcd87a3203fb6a1f99b2984b8667`.
- Linux/amd64 candidate binary SHA-256:
  `b3961315288871fa6085bf3b75c784a825a88f2bc3dd0694ad5d6c590eddf895`.
- Candidate archive SHA-256:
  `948bad5271dca08b9445c387c5aea7f58f22add6b47b9071e1801a756aa71259`.
- Toolchain: Go 1.26.6, `CGO_ENABLED=0`, `-trimpath`.
- Trusted Gitea verification run 119 and release-candidate run 120 passed for
  the exact implementation commit. CI and an independent, network-disabled
  build each produced the same archive digest twice.
- The installed receiver accepted only a pinned Ed25519 host key, a dedicated
  forced-command deployment key, the exact `tend-receive-v1` protocol, two
  allowlisted service names, and separately repeated artifact digests. An
  attempted arbitrary SSH command was refused.
- Production source, Go caches, repository credentials, secret values, remote
  paths, and shell commands did not cross the transport boundary.

Adding this evidence changes VCS build metadata but not executable logic. The
signed preview tag and attached release evidence therefore identify the final
evidence-bearing candidate and its required last maintenance pass.

### Gamertan blue-green campaign

- Application source commit:
  `a7e54047d3dc11671824b6ecc8ed698a9dd04421`.
- Preview 27 artifact SHA-256:
  `edadef3a97c089771e2b6bb7dadd58928762284878f76f23b112c61ae282f17f`.
- The artifact was built twice, byte-identically, in the pinned Go 1.26.6
  image with networking disabled and the audited cached module graph.
- The inactive slot passed health, readiness, page-marker, Caddy validation,
  public-origin, Sandwich Hime mount, and EQL continuity checks before traffic
  moved.
- Explicit rollback restored preview 26; its readiness and public boundary
  passed; preview 27 was then reactivated through the recorded state.
- Sandwich Hime's release pointer and service remained unchanged throughout.

### Sandwich Hime singleton campaign

- Application source commit:
  `435880c6751b773b6c5ee3ae6833d26e8eb7c0df`.
- Preview 30 artifact SHA-256:
  `bee8d5bcde2c3c2e8bb96d5909a19889fc7f0be9390046c243fe20c9ff2ca44b`.
- The artifact was built twice, byte-identically, in the same pinned,
  network-disabled Go 1.26.6 environment.
- A transient DynamicUser candidate passed health, readiness, tutorial marker,
  canonical-origin, and Gamertan-mounted checks before the singleton pointer
  and installed service changed. The candidate port was released afterward.
- Explicit rollback restored preview 28; its readiness and public boundary
  passed; preview 30 was then reactivated through the recorded state.
- Both Gamertan slots, its selected Caddy upstream, and EQL remained healthy.

### Findings resolved by dogfood

The first restricted transfer stopped before artifact validation because
`sudo` removed `SSH_ORIGINAL_COMMAND`. The forced account still refused the
request; no service state changed. The 0BSD sudoers template now preserves only
that one server-supplied variable, while the receiver requires its exact
protocol value and rejects every other command.

The first singleton candidate stopped before pointer mutation because the
shared environment file's live listen value overrode the candidate address.
The site remained on its former release and retained the same process. Tend now
rejects singleton shared environment files containing the configured listen
key; installed units own the non-secret live address and Tend supplies only the
transient candidate address.

Final verification found both service states valid, every installed unit and
Caddy active with zero restart failures, the candidate port closed, no warning
or error entries after the successful campaign, and representative Gamertan,
Sandwich Hime, mounted, and EQL routes returning HTTP 200.

## Assessed candidate

- Source commit: `306d085e518cb4fe7b20a66d1e2ceb171e54ebdc`.
- Linux/amd64 candidate binary SHA-256:
  `a9d53e286317d5acad9c0c321dc8d6240efee1e992714a892aba5be7c190dffc`.
- Candidate archive SHA-256:
  `52639d16cd55b1dfe7c4ce63d4523676a6e8fe7cf57ef25787938c4070f49bd3`.
- Toolchain: Go 1.26.6, `CGO_ENABLED=0`, `-trimpath`.
- Two fresh packages from the clean pushed commit were byte-identical.
- Tests, race tests, vet, license checks, public-snapshot isolation, and
  `govulncheck v1.1.4` passed. The vulnerability scan reported no known
  reachable vulnerabilities with the August 14, 2026 database.
- Trusted Gitea verification completed successfully for the exact commit.

The release tag and attached assets must still identify their own exact source
commit and digests. Any code change after this campaign requires the dogfood
sequence to be repeated.

A fresh-install check after the first immutable source tag found that the CLI
reported its development identity instead of the tagged module version. No
deployment logic or artifact content was ambiguous, but the distribution
identity was not acceptable. Preview 1 remains immutable and withdrawn;
Preview 2 adds Go build-information version selection and repeats the release
gates rather than retagging old content.

## Sandwich Hime website

The singleton-candidate strategy packaged and activated website preview 25:

- application source commit:
  `0429e3f0160aa4fd4d262bc5857bc232c2149cb8`;
- artifact SHA-256:
  `25025a05bb1aa6689f5f0779064b24a7c8193ff93c1395a2b3bd342588c3926a`;
- application toolchain: Go 1.26.6;
- isolated transient candidate passed health, readiness, and application
  smoke checks before the singleton pointer changed;
- explicit rollback returned to preview 24, and the exact approved preview 25
  artifact was then reactivated;
- canonical, documentation, news, `llms.txt`, and Gamertan-mounted routes
  returned HTTP 200 after reactivation.

## Gamertan

The blue-green strategy packaged and activated Gamertan preview 8:

- application source commit:
  `3acfa6a8e66ca3827c840d1fe9bc0b51c69c0a45`;
- artifact SHA-256:
  `76db7a9a6c496c204f653dc5e42c935272159320cc344b6a4db6374535c696ad`;
- application toolchain: Go 1.26.6;
- the inactive slot passed health, readiness, and page-marker checks before a
  validated atomic Caddy handler replacement;
- the handler retained `root:caddy` ownership, mode `0640`, and Sandwich Hime
  routing precedence;
- explicit rollback restored preview 7, and the exact approved preview 8
  artifact was then reactivated;
- homepage, project pages, news, feed, discovery files, the Sandwich Hime
  mount, and EQL Helper continuity returned HTTP 200 after reactivation.

## Finding resolved during the campaign

An earlier rollback attempt stopped safely before changing traffic because it
applied the new release's content markers to an older release whose route set
was different. Tend now uses the full configured smoke suite for new
deployments and health/readiness checks for an already-recorded rollback
target. A regression test requires that separation. The fixed candidate then
completed both live rollback sequences.

## Boundaries

The campaign covered one Linux/systemd/Caddy host and two small Go services.
It did not cover databases, migrations, containers, Kubernetes, hostile root,
or EQL Helper's application-specific catalog activation. Artifact transport
remains an application-owned, host-key-verified step outside Tend v0.1.