tend/docs/DOGFOOD_FRICTION.md

# Dogfood friction ledger

This document records friction observed through August 19, 2026 while using
Tend for real maintenance releases. Friction is evidence about the product: it
should become either a clearer contract, safer automation, or an explicit
non-goal. It must not become application-specific shell lore.

The findings below are maintainer observations, not implemented promises. Tend
continues to fail closed while an option is being designed. In particular,
there is no `--skip-remote` packaging escape hatch, no arbitrary deployment
hook, and no relaxation of the host-wide activation lock.

## Design standard

A Tend workflow should make the secure path the short, documented path:

- source, artifact, approval, configuration, and active-release identities are
  explicit and inspectable;
- credentials are narrowly scoped, briefly available, and absent from
  artifacts and logs;
- application validation is declarative, bounded, and cannot become a shell;
- first installation, maintenance, activation, rollback, and pruning are
  distinct operations;
- every mutation has a recorded prior state and a tested restoration path;
- Tend-owned state is an evidence journal and reconciliation aid, not an
  exclusive claim over an otherwise conventional systemd/Caddy service;
- tool-owned transient resources are leased, inspectable, resumable, and safe
  to reconcile without requiring operators to understand internal names;
- unrelated services may build and prepare candidates concurrently, while the
  shared Caddy activation boundary remains serialized.

## Observed findings

| ID | Status | Finding | Current safe behavior | Candidate direction |
| --- | --- | --- | --- | --- |
| F-01 | Workflow mitigation | CI packaging needs proof that the exact commit was pushed. | Packaging verifies the configured remote and fails if Git cannot authenticate. The trusted workflow retains only its job-scoped read-only checkout credential. | Make the proof credential or an authenticated source attestation an explicit Tend input and evidence boundary. |
| F-02 | Open | Application-owned unit and configuration changes are outside the binary activation transaction. | Operators stage, validate, back up, and restore those files separately. | Add an allowlisted configuration transaction or record and validate exact configuration digests. |
| F-03 | Open | A singleton candidate may not reproduce the installed unit's arguments and application configuration. | It receives the production environment file and an isolated listen override; activation still fails closed. | Add a bounded application preflight and explicit, validated candidate invocation. |
| F-04 | Partially resolved in development | Artifact production, review, approval, transfer, and activation require several identity checks. | v0.2 standardizes the evidence bundle and restricted transfer; `tend inspect` now exposes the complete non-mutating artifact validation as a named workflow. | Preserve explicit approval while adding digest-bound publication receipts. |
| F-05 | Partially resolved | Tend maintenance assumes an existing adopted service and current release. | `check-server` validates a prepared host, while first installation remains a separately reviewed operator procedure. | Define an explicit `install` or `adopt` transaction rather than silently treating bootstrap as maintenance. |
| F-06 | Partially resolved | Binary rollback can be unsafe when service configuration has changed incompatibly. | v0.2 records desired, candidate, active, previous, and last-attempt releases, but external configuration compatibility is operator-owned. | Bind non-secret configuration identities and preflight results to release state. |
| F-07 | Open | `GOPROXY=off` does not prove that every module metadata lookup is available locally. | Packaging stops before artifact creation when Go cannot resolve the complete pinned module graph. | Separate checksum-verified dependency resolution from a network-disabled, cache-completeness-checked build stage. |
| F-08 | Partially resolved in development | Packaging an otherwise clean pushed commit from a linked Git worktree fails Go's required VCS-status stamp. | Tend now detects a linked worktree before remote proof and build work and gives an exact standalone-clone instruction. | Assess provenance-preserving linked-worktree support without weakening `-buildvcs=true`. |
| F-09 | Resolved in Preview 2 | Restarting a fixed-address singleton briefly exposed Caddy to an unavailable upstream. | Tend now routes the imported handler to the candidate, restarts and proves the installed unit behind that handoff, then routes back while the candidate remains healthy through the activation window. | Preserve the production regression campaign and failure-injection matrix. |
| F-10 | Resolved in Preview 2 | A hardened root umask could make an extracted application binary executable only by root. | Tend reapplies every validated archive mode explicitly; extraction is tested under umask `0077`. | Preserve the mode and non-root candidate tests. |
| F-11 | Evidence practice | A single external observer can report common-mode client or network errors as apparent multi-service downtime. | Production campaigns retain server state and use an independent observer before classifying a continuity failure. | Standardize multi-vantage continuity evidence without making an observability dependency part of deployment authority. |
| F-12 | Partially resolved in development | A deliberately retained singleton handoff candidate could occupy Tend's fixed transient-unit name and block the next valid activation. | Candidates now have operation-scoped unit identities and bounded persisted leases; `reconcile --json` reports release identity, unit, pointer, and handler-file facts without mutation. | Add an explicit, idempotent `resume`/repair operation only after its route-identity and failure-injection model is proven. |
| F-13 | Open | A configured public-origin probe may resolve to a different deployment after a DNS or edge migration. | Operators separately verify local routed-origin identity and external DNS topology before activation. | Bind probes to expected release and edge identities; classify topology drift instead of treating a body marker as deployment proof. |
| F-14 | Open | An application can bind durable data identity to an absolute path that a binary-only candidate never exercises. | Preserve the production path and run application-owned validation before traffic moves. | Add a bounded, explicitly read-only application preflight and record which configuration/data identities it assessed. |
| F-15 | Product direction | Requiring Tend-specific residue and metadata to be perfect can make the facilitator feel like an exclusive deployment owner. | Conventional systemd/Caddy recovery remains authoritative and every manual intervention is captured as evidence. | Make state append-only and migratable, infer observed state safely, and keep Tend removable without making the service obscure or undeployable. |
| F-16 | Open | A reusable Docker network alias can identify both the live service and a retained handoff candidate, making the routed target ambiguous during activation. | Pin the temporary Caddy handoff to the exact, observed candidate address and restore the reviewed application handler after activation. | Allocate operation-scoped network identities and reject any candidate or handler target that resolves to more than one container or release identity. |
| F-17 | Open | A CI job can report a successful artifact upload while the forge artifact API exposes no retrievable artifact to the approval/deployment client. | Reproduce and verify the exact pushed commit locally, then compare two packages byte-for-byte before approving the digest; never guess an artifact URL. | Define a digest-bound artifact handoff with an independently readable receipt and fail the workflow unless the approval client can retrieve and inspect the exact uploaded bytes. |
| F-18 | Open | Tend's current strategies do not model a Docker Compose singleton, so a manual Observatory replacement briefly exposed an unavailable upstream. | The exact image, data backup, rollback image, and health probes were preserved, but Compose replaced the only live container and an external observer saw a bounded `502`/`503` window. | Design an explicit Compose strategy with operation-scoped candidates, unambiguous routing, durable-data preflight, activation continuity, and automatic rollback; do not add arbitrary container hooks to the systemd strategies. |
| F-19 | Open | A successful application schema migration can make the previous binary unreadable even when its image and service definition are intact. | Preserve verified database backups and prefer completing the proven forward activation; do not execute a binary-only rollback after a forward-only migration. | Make applications declare schema compatibility and an explicit data restoration or forward-recovery plan before Tend offers automatic rollback. |
| F-20 | Open | A stateful migration can require bounded temporary scratch space that a stateless binary candidate never exercises. | Run the exact candidate against a copied production data set under its proposed resource limits; record the reviewed scratch budget in the service definition. | Bind resource/configuration digests to the candidate and add an application-owned stateful preflight contract without arbitrary hooks. |

## F-01: pushed-commit proof in CI

### Observation

During an Observatory release-candidate run, verification intentionally checked
out source without persisted credentials. Tend later attempted to prove that
`HEAD` existed on the configured private Gitea branch and Git could not
authenticate. Packaging stopped before producing an artifact. This was the
correct fail-closed outcome, but the credential lifecycle was not obvious from
the workflow contract.

The trusted workflow uses a job-scoped read-only checkout credential so Tend
can perform the independent remote proof. It is not a long-lived repository or
deployment credential, and it is removed by checkout cleanup at the end of the
job. Production credentials remain unavailable to the build job.

### Options to assess

1. Support an explicit read-only Git credential file or credential helper for
   package-time proof. Never accept a token in arguments, configuration,
   manifests, artifacts, or logs.
2. Accept an authenticated CI source-attestation document binding repository,
   branch, commit, tree, workflow identity, and event identity. Define which
   CI issuers are trusted and preserve the attestation with release evidence.
3. Accept a signed, pre-verified source bundle whose identity and policy can be
   checked without network access.

Do not add a generic skip flag. An unavailable proof must remain a packaging
failure unless an equally strong proof mode was selected explicitly.

## F-02: application configuration is not binary activation

### Observation

Tend can transact an immutable binary, release pointers, a systemd restart, a
validated Caddy handler, health checks, and rollback state. It does not
currently transact application configuration, systemd unit changes, credential
bindings, or server-local secret files. A first deployment that changes these
files therefore needs a separate backup, validation, installation, and
restoration procedure.

This boundary is safe but easy to overlook: restoring the previous binary does
not restore an incompatible unit or application configuration.

### Options to assess

1. Add strict managed-file entries with exact source and destination paths,
   content digest, owner, group, mode, and validation type. Permit only regular
   files beneath configured roots; reject symlinks and unknown destinations.
2. Add a separate `tend configure` transaction that backs up, atomically
   replaces, validates, and restores supported systemd/Caddy/application files.
3. Keep configuration externally managed, but require Tend to record desired
   and active configuration digests and prove candidate/rollback compatibility.

None of these options should permit arbitrary shell commands. Secret values
remain referenced server-side and must never enter release artifacts or state.

## F-03: candidate fidelity

### Observation

A hardened singleton candidate receives the configured production environment
file and Tend's isolated listen-address override. It does not currently model
the installed systemd unit's complete argument vector. An application whose
configuration path is supplied by unit arguments may therefore start a
candidate with defaults instead of the intended production configuration.
That preserves live-state isolation, but it may prove only executable startup.
A missing credential, incompatible configuration field, filesystem permission,
or data migration requirement can then surface at activation time.

### Options to assess

- Define a fixed application preflight command or protocol that validates the
  production configuration, credentials, permissions, and data compatibility
  without binding the live port or mutating live state.
- Add a strict candidate argument vector to schema 2. Validate each argument,
  reject secret values and paths outside the application contract, and pass it
  directly to systemd without a shell.
- Permit an allowlisted `check` argument vector, never a shell string, with
  bounded time/output and an explicitly non-mutating application contract.
- Record which checks ran against the binary alone and which ran against the
  actual production configuration so the evidence cannot overstate coverage.

## F-04: artifact review and approval ergonomics

### Observation

The secure flow deliberately separates build, review, digest approval,
transfer, candidate validation, and activation. v0.2 now produces a consistent
archive, manifest, SBOM, and checksum set, then transfers one exact artifact
through its restricted receiver. In practice, operators still need a
predictable way to discover that CI artifact, inspect it, approve exactly one
digest, and retain the review evidence. Without a first-class review path,
correct manual steps are easy to reconstruct differently for each application.

### Options to assess

- Add `tend inspect` for offline, non-mutating verification and a concise human
  and JSON summary of source, build, dependency, and archive identities.
- Add an approval record that binds the artifact digest, service, target,
  approver, and expiry without containing a credential.
- Standardize one Gitea artifact layout and documented download-to-activation
  workflow. Keep production credentials unavailable to verification jobs.

Approval must remain explicit; better ergonomics must not turn a successful
build into an automatic production mutation.

## F-05: first installation and adoption

### Observation

The maintenance workflow expects an installed service, valid configuration,
and a current release pointer. `check-server` validates the restricted receiver
policy and prepared service boundary, but it does not create them. Observatory
bootstrap therefore required operator-managed service account, directories,
credentials, unit, configuration, and an initial current release before Tend
could own later maintenance safely.

### Options to assess

- `tend install`: a deliberately broader, separately approved transaction with
  a strict schema and complete rollback of every supported created object.
- `tend adopt`: validate an existing service and release, copy or identify its
  immutable artifact, establish state, and refuse ambiguous ownership.
- Keep bootstrap out of Tend, but ship a versioned acceptance checklist and a
  machine-readable `check-server` result that maintenance can require.

Installation and adoption must not be inferred from a missing state file.

## F-06: release and configuration identity

### Observation

Content-addressed releases and v0.2 state make desired, candidate, active,
previous, and last-attempt binary identities clear. A live service is still a
combination of its binary, application configuration, credential bindings,
unit, routing fragment, and sometimes data schema. Tend cannot yet fully
explain that combined identity or determine whether a retained binary is
compatible with the current external configuration.

### Options to assess

- Extend the existing desired, candidate, active, previous, and last-attempt
  release state with non-secret configuration and unit digests.
- Require rollback compatibility declarations or read-only application
  preflight before changing traffic.
- Expose the identities and last validation results through
  `tend status --json` for deployment evidence and future Observatory
  ingestion.

Configuration records contain digests and approved metadata only—not secret
contents.

## F-07: offline module-cache completeness

### Observation

An exact Observatory package attempt used `GOPROXY=off` and a previously used
module cache. Source archives for the application dependencies were present,
but Go still needed several module metadata records while Tend enumerated the
complete build graph. Go refused the lookup and Tend stopped before building
or writing an artifact. Re-enabling the checksum-verified public proxy supplied
the missing metadata; the resulting package was byte-identical to the trusted
CI candidate.

This is safe failure, but `GOPROXY=off` alone is not evidence of a hermetic
build. A cache can be partially populated even when ordinary builds happen to
succeed.

### Options to assess

- Add a resolver stage that runs with the pinned toolchain, proxy, and checksum
  database, emits the complete module inventory, and materializes a bounded,
  read-only cache for the builder.
- Run Tend's package stage with networking disabled and require every module,
  checksum, source archive, and metadata record to come from that reviewed
  cache.
- Add a non-mutating cache-completeness check that reports missing module
  identities before the expensive double build.
- Consider a strictly verified vendored-source mode where repository size and
  update review are acceptable; do not silently change dependency modes.

Do not treat `GONOSUMDB`, `GOPRIVATE`, or a proxy bypass as an offline-build
solution. They alter verification or routing policy rather than proving cache
completeness.

## F-08: linked-worktree VCS stamping

### Observation

An exact clean, pushed Observatory main commit passed verification and public
snapshot isolation from a detached linked Git worktree. Tend accepted its
source and remote provenance, then Go 1.26.6 stopped both `tend package` and an
equivalent direct build at the required `-buildvcs=true` step with `error
obtaining VCS status: exit status 128`. Ordinary Git status and commit queries
from the same worktree succeeded.

A fresh standalone SSH clone of the identical commit packaged successfully
twice. Both archives were byte-identical and carried the expected commit, Go
version, VCS settings, checksums, manifest, and SBOM. This isolates the failure
to the linked-worktree build shape rather than the application source or module
graph.

The failure is safe: Tend did not create a partial artifact and must not switch
to `-buildvcs=false`, because the package gate independently verifies the
embedded VCS revision and clean state. Until the interaction is resolved, use
a clean standalone clone for release packaging.

### Options to assess

- Detect a `.git` indirection file during `tend check` and fail before the
  expensive double build with a precise standalone-clone instruction.
- Reproduce the interaction in a package integration test against the minimum
  supported Go release and determine whether it is a Go toolchain limitation
  or an invocation/environment defect.
- If linked worktrees can be supported, require the resulting build record to
  carry the exact expected `vcs.revision` and `vcs.modified=false`; do not
  synthesize those settings or disable VCS stamping.
- Consider an explicit, signed source-bundle input as part of the broader
  source-attestation design. It must remain at least as strong as current
  pushed-commit proof.

## F-09: singleton traffic continuity

### Observation

During the August 18 Observatory maintenance exercise, the transient candidate
passed health, readiness, and content checks. Tend then stopped that candidate,
changed the singleton pointer, and restarted the installed fixed-address unit.
Caddy still targeted the fixed address during that restart, so the Observatory
agent observed a small number of transient HTTP `502` responses. Its durable
spool retried successfully and no accepted telemetry was lost, but the routed
origin was not continuously available.

The corrected activation contract treats the candidate as a traffic handoff,
not merely a preflight process. Tend validates and routes the imported Caddy
handler to the candidate before changing the release pointer. It restarts and
probes the fixed-address unit without public traffic, routes back only after
that unit passes, and keeps the candidate healthy throughout the bounded
activation window. Any failure restores the former pointer and exact handler
bytes. If restoration itself cannot be completed, Tend leaves the proven
candidate routed and running for explicit operator recovery instead of causing
a known outage.

Regression tests cover successful handoff, restart failure, public-origin
failure during the activation window, pointer and handler restoration, Caddy
validation/reload boundaries, and candidate cleanup. Final production evidence
must still repeat deploy, rollback, and reactivation with the exact release
binary before this finding is treated as released.

## F-10: archive modes under a hardened umask

### Observation

The first two August 18 maintenance candidates stopped before activation. The
release archives correctly recorded executable mode `0755`, but extraction by
root under umask `0077` left the installed application binary mode `0700`.
The unprivileged transient candidate could not execute it. Tend emitted failed
candidate evidence, retained the prior release and route, and did not expose
the failed binary to traffic.

Extraction now reapplies the already validated archive mode after file content
is closed. A Linux regression test sets umask `0077`, extracts the release, and
requires the installed binary to remain executable by the service identity.
The successful maintenance campaign used that corrected Tend binary.

## F-12: retained-candidate reconciliation

### Observation

An Observatory activation failure intentionally left its proven handoff
candidate routed and running for operator recovery. A later push transferred
and validated a new content-addressed artifact, then stopped before candidate
startup because systemd still had Tend's fixed candidate unit loaded:

`Unit observatory-tend-candidate.service was already loaded or has a fragment file.`

The artifact, application, and host were valid. The collision was Tend-owned
residue from its own safe fallback behavior. Recovery required starting the
same stateless fallback under an independently named hardened unit, proving it,
routing Caddy to it, and only then stopping the old candidate and confirming
its port and unit name were free. No accepted application state was lost, but
the operator had to understand Tend's internal unit convention.

### Implemented foundation

- Transient candidate units are named by service plus the first 12 hexadecimal
  characters of a fresh 128-bit operation identity.
- Conventional deployment state remains schema 1 so the retained v0.1 binary
  can still read active and previous release identities. Singleton operations
  write a separate strict, adjacent candidate-lease file containing the
  operation, release, unit, address, and start time.
- `tend reconcile --json` reports persisted state, installed/candidate unit
  activity, release pointers, and whether the exact handler file matches the
  installed or candidate upstream. It explicitly distinguishes handler file
  bytes from Caddy's loaded runtime configuration and performs no mutation.
- A retained forward-activation or rollback candidate lease survives an
  incomplete recovery instead of being erased by a generic failed-attempt path.
  Older binaries ignore the additive lease file, so operators must reconcile
  before downgrading or attempting another activation.

### Remaining options to assess

- Extend the candidate lease with separately observed health and loaded-route
  evidence rather than inferring either from a successful file write.
- Add an explicit `tend resume` operation that distinguishes active, routed,
  rollback, abandoned, and unknown candidates without mutating by default.
- Permit automatic cleanup only after Tend proves that Caddy, current/previous
  pointers, and the installed unit do not reference the candidate and that a
  healthy route remains.
- Make a repeated activation of the same approved digest idempotently resume
  the recorded operation instead of restaging or colliding with itself.

Never resolve the collision by blindly stopping the loaded unit. A retained
candidate may be the only healthy route after a failed singleton activation.

## F-13: public-origin topology and release identity

### Observation

During the same recovery, the public DNS record had already moved to a new
edge while the old host's Tend policy still named the canonical HTTPS origin
as its post-activation smoke target. A request from the old host would therefore
test a different deployment. It could fail even when the old host was healthy,
or pass against a matching marker served by the new host. Neither result proves
the release that Tend just activated.

### Options to assess

- Separate a local routed-origin probe—Caddy with the canonical Host/SNI on the
  deployment host—from an external DNS-origin observation.
- Bind local success to a non-secret application release identity such as the
  approved artifact digest, commit, and version, not only a human page marker.
- Record the expected public edge identity or resolved address set at approval
  time. If it changes, classify the result as topology drift and require an
  explicit migration decision instead of reporting an application failure.
- Allow independent external observers as additional evidence, but never let
  a response from an unidentified deployment authorize activation.

Public reachability remains valuable; it must be evidence about the intended
deployment rather than merely evidence that the hostname answered.

## F-14: application data-path and preflight fidelity

### Observation

An exact, checksummed Observatory data copy was mounted at a different absolute
path in a container. Raw objects and durable SQLite databases were byte
identical, but the application catalogue intentionally compared stored segment
paths with filesystem-derived identities and failed closed. Preserving the
original in-container data path made the same Preview 15 binary ready in four
seconds with zero restarts and bounded memory.

This was an application portability constraint, not a reason for Tend to
rewrite database state. It also demonstrates the limit of a binary-only
stateless candidate: executable health cannot prove compatibility with the
real configuration, credentials, mounts, or durable data.

SQLite `-wal`, `-shm`, and process-lock files also changed during ordinary
open/close behavior while raw segments and durable databases did not. Migration
evidence should distinguish durable application truth from ephemeral runtime
coordination files.

### Options to assess

- Support a fixed, bounded application-owned preflight argument vector with an
  explicit non-mutating contract, timeout, output limit, and no shell.
- Record the exact non-secret configuration digest, data-root identity, mount
  identity, and checks performed alongside binary candidate evidence.
- Let applications define their durable migration evidence set; Tend should
  transport and report those digests but must not infer database semantics or
  edit state.
- Require rollback compatibility to be assessed against the same configuration
  and data identities before an older binary is called safe.

## F-15: facilitator rather than owner

Tend's strict boundaries are valuable where authority changes hands: source
provenance, artifact digest approval, restricted transport, secret references,
path validation, traffic movement, and rollback. Strictness becomes harmful
when Tend's internal names or stale metadata are treated as application
requirements that an authorized maintainer must reverse-engineer.

The target model is less invasive:

- systemd units, Caddy handlers, environment files, release directories, and
  application checks remain conventional and independently operable;
- Tend records an append-only migration/deployment journal and derives an
  observed state before proposing mutation;
- schema migrations preserve old records and explain compatibility rather than
  silently rejecting safe, recognizable state;
- `check`, `status`, and `reconcile` show facts and proposed repairs without
  mutation; approval is required for artifact selection and traffic movement,
  not for Tend implementation trivia;
- installation and adoption are explicit, while uninstalling Tend leaves an
  understandable, runnable service and complete evidence trail.

This does not relax hostile-input defenses. It moves strictness to the trust
boundary and makes recovery humane for the authenticated operator.

## F-16: ambiguous container-network identity

### Observation

During the Observatory Preview 16 activation, the retained stateless handoff
container and the live Compose service both answered to the Docker network
alias `observatory`. A Caddy target such as `observatory:8093` could therefore
resolve to either release. Removing or restarting one container could also
change which release received traffic. The application and both containers
were healthy; the ambiguity existed solely in deployment identity.

The activation used a separately validated Caddy fragment pinned to the exact
candidate address, proved its release marker before traffic moved, replaced
the live service, and then restored the reviewed application handler. No
ambiguous alias was used for the handoff.

### Options to assess

- Give every candidate an operation-scoped container name, network alias, and
  lease that bind directly to its artifact digest and expected address.
- Resolve and inspect a proposed upstream immediately before Caddy validation;
  reject zero, multiple, or identity-mismatched targets.
- Prefer a dedicated candidate network or an address supplied by the container
  runtime over a stable alias shared with the live service.
- Record the exact routed target and release proof in activation state, then
  verify that the restored production handler identifies the active release.

Do not treat a healthy response from an ambiguous service name as release
proof. Availability and deployment identity are separate properties.

## F-17: CI artifact publication is not artifact availability

### Observation

The trusted Observatory release-candidate workflow completed its two builds,
byte comparison, checksums, manifest, and SBOM, and its upload action reported
success. The forge artifact API subsequently returned no artifact for that
run. The deployment client therefore had no independently discoverable object
to download and inspect. Guessing a web-interface route or treating a green
upload step as possession of the bytes would have weakened the build-approve-
deploy boundary.

The release was instead reproduced twice from a fresh, clean clone of the
exact pushed commit with the pinned toolchain. The packages compared
byte-for-byte, their manifest, checksums, SBOM, version, commit, and tree were
verified, and the approved digest was recorded. This preserved release
identity, but it is an operator workaround rather than the desired CI handoff.

### Options to assess

- Require the publisher to return a versioned receipt containing forge, run,
  artifact identifier, size, digest, retention, and retrieval endpoint.
- Add a separate read-only verification step that downloads the artifact using
  the same interface available to the approval client and reruns `tend
  inspect` before the workflow is considered publish-complete.
- Support a content-addressed, append-only artifact store whose object name is
  the approved SHA-256 and whose credentials remain separate from production.
- Let `tend push` accept only a locally present artifact plus an optional
  verified publication receipt; never allow a successful CI status alone to
  select bytes for deployment.

Artifact availability must be proven from the consumer side. A successful
upload log is evidence of an attempted publication, not evidence that the
approved bytes can be recovered.

The same failure repeated for Tend release-candidate run 369: the pinned job
successfully built identical archives, recorded digest
`2c225e0b9dd0be2d36fda62ab8d0b52cd9b28f9336c60dfb5edf3e715f450f95`,
and finalized a 3.18 MB upload, while both repository-wide and run-scoped REST
artifact listings returned no objects. An independent clean build reproduced
the exact digest before dogfooding. This confirms the problem is a reusable
forge-to-approval gap rather than an Observatory-specific packaging defect.

## F-18: Docker Compose is not a systemd singleton

### Observation

Observatory Preview 18 was packaged reproducibly and its exact scratch image
passed a constrained loopback candidate check. The production service on
cliff-mads is a Docker Compose singleton, while Tend's supported strategies are
systemd/Caddy blue-green and systemd singleton-candidate. Tend was therefore
not used to pretend that an unsupported runtime had received a complete Tend
activation proof.

The manual Compose replacement retained the former image and online database
backups and completed health, readiness, route, projection, agent, and log
checks. An independent public observer nevertheless recorded `200`, then
`502`/`503`, then `200` across an approximately 31.6-second replacement and
startup window. No application restart loop or accepted-data loss was
observed, but this is not continuous delivery.

Preview.19 strengthened this finding. The exact candidate passed direct health
and readiness, but an activation script treated the first transient routed
`503` as final and attempted rollback before the proxy path had settled. The
previous binary then correctly rejected the newly migrated control schema and
entered a fail-closed restart loop. The operator recovered with the already
proven forward candidate while the agent remained paused. This was a bounded,
observable failure with intact backups, but it demonstrates that proxy
settling and data-schema rollback are different gates and cannot share one
generic failure branch.

### Options to assess

- Add a distinct, versioned Docker Compose strategy rather than arbitrary
  command hooks or hidden container behavior inside existing strategies.
- Allocate an operation-scoped candidate container and network identity that
  cannot collide with the production alias.
- Validate the exact image digest, non-secret Compose/configuration digest,
  mounts, durable data identity, and application-owned read-only preflight
  before moving traffic.
- Route Caddy to the proven candidate, replace or promote the production
  service while the candidate remains healthy, then restore the reviewed live
  handler only after the final container proves its release identity.
- Restore the prior handler and retained image automatically when any
  activation-window probe fails; keep data rollback an explicit
  application-owned decision.

This strategy must remain optional. Tend should facilitate a conventional
Compose service without requiring Tend-only container labels, aliases, or
state for ordinary operator recovery.

## F-19: binary rollback is not data rollback

### Observation

Observatory Preview.19 migrated its live control database from the schema
understood by Preview.18 to schema 11 before activation. When a premature
public-origin failure triggered binary rollback, Preview.18 rejected the new
schema rather than interpreting unknown state. That fail-closed behavior was
correct, but Tend-like orchestration cannot infer from two binary identities
whether their durable schemas are mutually readable.

The exact pre-migration control and projection databases had already been
copied, integrity-checked, mode-restricted, and SHA-256 verified. The new
candidate had also passed the migration on an isolated copy. Because the
migration was forward-valid and the old agent was paused, completing the
Preview.19 activation preserved more verified state than restoring the backup.

### Options to assess

- Let an application declare a versioned, bounded compatibility statement for
  current-to-candidate and candidate-to-previous data access.
- Treat migration completion as a journalled phase after which automatic
  binary rollback is permitted only when backward readability was explicitly
  proven.
- Support an application-owned, separately approved data-restoration plan;
  never infer one from release pointers or run an arbitrary rollback hook.
- Keep verified backups and the proven forward candidate until the migration
  soak closes, even when the previous binary remains retained.

## F-20: stateful preflight needs its real resource envelope

### Observation

The Preview.19 stateless candidate and ordinary tests passed, but building a
presence-only SQLite index over a copied production projection failed with
`database or disk is full` under the service's 64 MiB `/tmp` tmpfs. The exact
same candidate and data passed with a 512 MiB tmpfs and drained all pending raw
segments. The projection database was healthy; the one-time index sort needed
more bounded scratch space than steady-state operation.

### Options to assess

- Record the non-secret service configuration digest and resource envelope
  beside the artifact digest.
- Permit a strict application-owned preflight command selected from a reviewed
  schema, never an arbitrary shell string.
- Exercise migrations against a copied or snapshot-backed production data set
  under the candidate's exact CPU, memory, temporary-storage, filesystem, and
  credential boundaries.
- Report resource exhaustion distinctly from corrupt data, failed health, or
  incompatible schema so recovery guidance remains accurate.

## Prioritization

The recommended implementation order is:

1. operation-scoped candidate leases plus explicit resume/repair built on the
   new read-only reconciliation report;
2. migration compatibility and stateful-resource preflight contracts;
3. a bounded Docker Compose strategy with operation-scoped container/network
   identities and unambiguous upstream
   resolution and release proof;
4. release-bound local routed-origin proof separated from external DNS proof;
5. a bounded application preflight and configuration/data identity record;
6. a consumer-verifiable, digest-bound CI artifact publication receipt;
7. final production proof of the singleton traffic handoff;
8. a checksum-verified resolver and network-disabled package contract;
9. a provenance-preserving linked-worktree support decision after the new
   early diagnostic;
10. a standardized CI artifact/approval contract building on `tend inspect`;
11. exact release plus configuration identity in status and state;
12. restricted transfer and receive with host policy;
13. explicit adoption for existing services;
14. managed configuration only after its restoration and failure-injection
   model is as strong as binary activation.

Friction entries should be updated with the implementing version, tests, and
dogfood evidence when resolved. Resolved entries remain in this ledger so the
reason for the security boundary is not lost.