tend/docs/DOGFOOD_FRICTION.md

Dogfood friction ledger

This document records friction observed through August 19, 2026 while using Tend for real maintenance releases. Friction is evidence about the product: it should become either a clearer contract, safer automation, or an explicit non-goal. It must not become application-specific shell lore.

The findings below are maintainer observations, not implemented promises. Tend continues to fail closed while an option is being designed. In particular, there is no --skip-remote packaging escape hatch, no arbitrary deployment hook, and no relaxation of the host-wide activation lock.

Design standard

A Tend workflow should make the secure path the short, documented path:

• source, artifact, approval, configuration, and active-release identities are explicit and inspectable;
• credentials are narrowly scoped, briefly available, and absent from artifacts and logs;
• application validation is declarative, bounded, and cannot become a shell;
• first installation, maintenance, activation, rollback, and pruning are distinct operations;
• every mutation has a recorded prior state and a tested restoration path;
• Tend-owned state is an evidence journal and reconciliation aid, not an exclusive claim over an otherwise conventional systemd/Caddy service;
• tool-owned transient resources are leased, inspectable, resumable, and safe to reconcile without requiring operators to understand internal names;
• unrelated services may build and prepare candidates concurrently, while the shared Caddy activation boundary remains serialized.

Observed findings

ID	Status	Finding	Current safe behavior	Candidate direction
F-01	Workflow mitigation	CI packaging needs proof that the exact commit was pushed.	Packaging verifies the configured remote and fails if Git cannot authenticate. The trusted workflow retains only its job-scoped read-only checkout credential.	Make the proof credential or an authenticated source attestation an explicit Tend input and evidence boundary.
F-02	Open	Application-owned unit and configuration changes are outside the binary activation transaction.	Operators stage, validate, back up, and restore those files separately.	Add an allowlisted configuration transaction or record and validate exact configuration digests.
F-03	Open	A singleton candidate may not reproduce the installed unit's arguments and application configuration.	It receives the production environment file and an isolated listen override; activation still fails closed.	Add a bounded application preflight and explicit, validated candidate invocation.
F-04	Partially resolved in development	Artifact production, review, approval, transfer, and activation require several identity checks.	v0.2 standardizes the evidence bundle and restricted transfer; tend inspect now exposes the complete non-mutating artifact validation as a named workflow.	Preserve explicit approval while adding digest-bound publication receipts.
F-05	Partially resolved	Tend maintenance assumes an existing adopted service and current release.	check-server validates a prepared host, while first installation remains a separately reviewed operator procedure.	Define an explicit install or adopt transaction rather than silently treating bootstrap as maintenance.
F-06	Partially resolved	Binary rollback can be unsafe when service configuration has changed incompatibly.	v0.2 records desired, candidate, active, previous, and last-attempt releases, but external configuration compatibility is operator-owned.	Bind non-secret configuration identities and preflight results to release state.
F-07	Open	GOPROXY=off does not prove that every module metadata lookup is available locally.	Packaging stops before artifact creation when Go cannot resolve the complete pinned module graph.	Separate checksum-verified dependency resolution from a network-disabled, cache-completeness-checked build stage.
F-08	Partially resolved in development	Packaging an otherwise clean pushed commit from a linked Git worktree fails Go's required VCS-status stamp.	Tend now detects a linked worktree before remote proof and build work and gives an exact standalone-clone instruction.	Assess provenance-preserving linked-worktree support without weakening -buildvcs=true.
F-09	Resolved in Preview 2	Restarting a fixed-address singleton briefly exposed Caddy to an unavailable upstream.	Tend now routes the imported handler to the candidate, restarts and proves the installed unit behind that handoff, then routes back while the candidate remains healthy through the activation window.	Preserve the production regression campaign and failure-injection matrix.
F-10	Resolved in Preview 2	A hardened root umask could make an extracted application binary executable only by root.	Tend reapplies every validated archive mode explicitly; extraction is tested under umask 0077.	Preserve the mode and non-root candidate tests.
F-11	Evidence practice	A single external observer can report common-mode client or network errors as apparent multi-service downtime.	Production campaigns retain server state and use an independent observer before classifying a continuity failure.	Standardize multi-vantage continuity evidence without making an observability dependency part of deployment authority.
F-12	Partially resolved in development	A deliberately retained singleton handoff candidate could occupy Tend's fixed transient-unit name and block the next valid activation.	Candidates now have operation-scoped unit identities and bounded persisted leases; reconcile --json reports release identity, unit, pointer, and handler-file facts without mutation.	Add an explicit, idempotent resume/repair operation only after its route-identity and failure-injection model is proven.
F-13	Open	A configured public-origin probe may resolve to a different deployment after a DNS or edge migration.	Operators separately verify local routed-origin identity and external DNS topology before activation.	Bind probes to expected release and edge identities; classify topology drift instead of treating a body marker as deployment proof.
F-14	Open	An application can bind durable data identity to an absolute path that a binary-only candidate never exercises.	Preserve the production path and run application-owned validation before traffic moves.	Add a bounded, explicitly read-only application preflight and record which configuration/data identities it assessed.
F-15	Product direction	Requiring Tend-specific residue and metadata to be perfect can make the facilitator feel like an exclusive deployment owner.	Conventional systemd/Caddy recovery remains authoritative and every manual intervention is captured as evidence.	Make state append-only and migratable, infer observed state safely, and keep Tend removable without making the service obscure or undeployable.
F-16	Open	A reusable Docker network alias can identify both the live service and a retained handoff candidate, making the routed target ambiguous during activation.	Pin the temporary Caddy handoff to the exact, observed candidate address and restore the reviewed application handler after activation.	Allocate operation-scoped network identities and reject any candidate or handler target that resolves to more than one container or release identity.
F-17	Open	A CI job can report a successful artifact upload while the forge artifact API exposes no retrievable artifact to the approval/deployment client.	Reproduce and verify the exact pushed commit locally, then compare two packages byte-for-byte before approving the digest; never guess an artifact URL.	Define a digest-bound artifact handoff with an independently readable receipt and fail the workflow unless the approval client can retrieve and inspect the exact uploaded bytes.
F-18	Open	Tend's current strategies do not model a Docker Compose singleton, so a manual Observatory replacement briefly exposed an unavailable upstream.	The exact image, data backup, rollback image, and health probes were preserved, but Compose replaced the only live container and an external observer saw a bounded 502/503 window.	Design an explicit Compose strategy with operation-scoped candidates, unambiguous routing, durable-data preflight, activation continuity, and automatic rollback; do not add arbitrary container hooks to the systemd strategies.
F-19	Open	A successful application schema migration can make the previous binary unreadable even when its image and service definition are intact.	Preserve verified database backups and prefer completing the proven forward activation; do not execute a binary-only rollback after a forward-only migration.	Make applications declare schema compatibility and an explicit data restoration or forward-recovery plan before Tend offers automatic rollback.
F-20	Open	A stateful migration can require bounded temporary scratch space that a stateless binary candidate never exercises.	Run the exact candidate against a copied production data set under its proposed resource limits; record the reviewed scratch budget in the service definition.	Bind resource/configuration digests to the candidate and add an application-owned stateful preflight contract without arbitrary hooks.

F-01: pushed-commit proof in CI

Observation

During an Observatory release-candidate run, verification intentionally checked out source without persisted credentials. Tend later attempted to prove that HEAD existed on the configured private Gitea branch and Git could not authenticate. Packaging stopped before producing an artifact. This was the correct fail-closed outcome, but the credential lifecycle was not obvious from the workflow contract.

The trusted workflow uses a job-scoped read-only checkout credential so Tend can perform the independent remote proof. It is not a long-lived repository or deployment credential, and it is removed by checkout cleanup at the end of the job. Production credentials remain unavailable to the build job.

Options to assess

1. Support an explicit read-only Git credential file or credential helper for package-time proof. Never accept a token in arguments, configuration, manifests, artifacts, or logs.
2. Accept an authenticated CI source-attestation document binding repository, branch, commit, tree, workflow identity, and event identity. Define which CI issuers are trusted and preserve the attestation with release evidence.
3. Accept a signed, pre-verified source bundle whose identity and policy can be checked without network access.

Do not add a generic skip flag. An unavailable proof must remain a packaging failure unless an equally strong proof mode was selected explicitly.

F-02: application configuration is not binary activation

Observation

Tend can transact an immutable binary, release pointers, a systemd restart, a validated Caddy handler, health checks, and rollback state. It does not currently transact application configuration, systemd unit changes, credential bindings, or server-local secret files. A first deployment that changes these files therefore needs a separate backup, validation, installation, and restoration procedure.

This boundary is safe but easy to overlook: restoring the previous binary does not restore an incompatible unit or application configuration.

Options to assess

1. Add strict managed-file entries with exact source and destination paths, content digest, owner, group, mode, and validation type. Permit only regular files beneath configured roots; reject symlinks and unknown destinations.
2. Add a separate tend configure transaction that backs up, atomically replaces, validates, and restores supported systemd/Caddy/application files.
3. Keep configuration externally managed, but require Tend to record desired and active configuration digests and prove candidate/rollback compatibility.

None of these options should permit arbitrary shell commands. Secret values remain referenced server-side and must never enter release artifacts or state.

F-03: candidate fidelity

Observation

A hardened singleton candidate receives the configured production environment file and Tend's isolated listen-address override. It does not currently model the installed systemd unit's complete argument vector. An application whose configuration path is supplied by unit arguments may therefore start a candidate with defaults instead of the intended production configuration. That preserves live-state isolation, but it may prove only executable startup. A missing credential, incompatible configuration field, filesystem permission, or data migration requirement can then surface at activation time.

Options to assess

• Define a fixed application preflight command or protocol that validates the production configuration, credentials, permissions, and data compatibility without binding the live port or mutating live state.
• Add a strict candidate argument vector to schema 2. Validate each argument, reject secret values and paths outside the application contract, and pass it directly to systemd without a shell.
• Permit an allowlisted check argument vector, never a shell string, with bounded time/output and an explicitly non-mutating application contract.
• Record which checks ran against the binary alone and which ran against the actual production configuration so the evidence cannot overstate coverage.

F-04: artifact review and approval ergonomics

Observation

The secure flow deliberately separates build, review, digest approval, transfer, candidate validation, and activation. v0.2 now produces a consistent archive, manifest, SBOM, and checksum set, then transfers one exact artifact through its restricted receiver. In practice, operators still need a predictable way to discover that CI artifact, inspect it, approve exactly one digest, and retain the review evidence. Without a first-class review path, correct manual steps are easy to reconstruct differently for each application.

Options to assess

• Add tend inspect for offline, non-mutating verification and a concise human and JSON summary of source, build, dependency, and archive identities.
• Add an approval record that binds the artifact digest, service, target, approver, and expiry without containing a credential.
• Standardize one Gitea artifact layout and documented download-to-activation workflow. Keep production credentials unavailable to verification jobs.

Approval must remain explicit; better ergonomics must not turn a successful build into an automatic production mutation.

F-05: first installation and adoption

Observation

The maintenance workflow expects an installed service, valid configuration, and a current release pointer. check-server validates the restricted receiver policy and prepared service boundary, but it does not create them. Observatory bootstrap therefore required operator-managed service account, directories, credentials, unit, configuration, and an initial current release before Tend could own later maintenance safely.

Options to assess

• tend install: a deliberately broader, separately approved transaction with a strict schema and complete rollback of every supported created object.
• tend adopt: validate an existing service and release, copy or identify its immutable artifact, establish state, and refuse ambiguous ownership.
• Keep bootstrap out of Tend, but ship a versioned acceptance checklist and a machine-readable check-server result that maintenance can require.

Installation and adoption must not be inferred from a missing state file.

F-06: release and configuration identity

Observation

Content-addressed releases and v0.2 state make desired, candidate, active, previous, and last-attempt binary identities clear. A live service is still a combination of its binary, application configuration, credential bindings, unit, routing fragment, and sometimes data schema. Tend cannot yet fully explain that combined identity or determine whether a retained binary is compatible with the current external configuration.

Options to assess

• Extend the existing desired, candidate, active, previous, and last-attempt release state with non-secret configuration and unit digests.
• Require rollback compatibility declarations or read-only application preflight before changing traffic.
• Expose the identities and last validation results through tend status --json for deployment evidence and future Observatory ingestion.

Configuration records contain digests and approved metadata only—not secret contents.

F-07: offline module-cache completeness

Observation

An exact Observatory package attempt used GOPROXY=off and a previously used module cache. Source archives for the application dependencies were present, but Go still needed several module metadata records while Tend enumerated the complete build graph. Go refused the lookup and Tend stopped before building or writing an artifact. Re-enabling the checksum-verified public proxy supplied the missing metadata; the resulting package was byte-identical to the trusted CI candidate.

This is safe failure, but GOPROXY=off alone is not evidence of a hermetic build. A cache can be partially populated even when ordinary builds happen to succeed.

Options to assess

• Add a resolver stage that runs with the pinned toolchain, proxy, and checksum database, emits the complete module inventory, and materializes a bounded, read-only cache for the builder.
• Run Tend's package stage with networking disabled and require every module, checksum, source archive, and metadata record to come from that reviewed cache.
• Add a non-mutating cache-completeness check that reports missing module identities before the expensive double build.
• Consider a strictly verified vendored-source mode where repository size and update review are acceptable; do not silently change dependency modes.

Do not treat GONOSUMDB, GOPRIVATE, or a proxy bypass as an offline-build solution. They alter verification or routing policy rather than proving cache completeness.

F-08: linked-worktree VCS stamping

Observation

An exact clean, pushed Observatory main commit passed verification and public snapshot isolation from a detached linked Git worktree. Tend accepted its source and remote provenance, then Go 1.26.6 stopped both tend package and an equivalent direct build at the required -buildvcs=true step with error obtaining VCS status: exit status 128. Ordinary Git status and commit queries from the same worktree succeeded.

A fresh standalone SSH clone of the identical commit packaged successfully twice. Both archives were byte-identical and carried the expected commit, Go version, VCS settings, checksums, manifest, and SBOM. This isolates the failure to the linked-worktree build shape rather than the application source or module graph.

The failure is safe: Tend did not create a partial artifact and must not switch to -buildvcs=false, because the package gate independently verifies the embedded VCS revision and clean state. Until the interaction is resolved, use a clean standalone clone for release packaging.

Options to assess

• Detect a .git indirection file during tend check and fail before the expensive double build with a precise standalone-clone instruction.
• Reproduce the interaction in a package integration test against the minimum supported Go release and determine whether it is a Go toolchain limitation or an invocation/environment defect.
• If linked worktrees can be supported, require the resulting build record to carry the exact expected vcs.revision and vcs.modified=false; do not synthesize those settings or disable VCS stamping.
• Consider an explicit, signed source-bundle input as part of the broader source-attestation design. It must remain at least as strong as current pushed-commit proof.

F-09: singleton traffic continuity

Observation

During the August 18 Observatory maintenance exercise, the transient candidate passed health, readiness, and content checks. Tend then stopped that candidate, changed the singleton pointer, and restarted the installed fixed-address unit. Caddy still targeted the fixed address during that restart, so the Observatory agent observed a small number of transient HTTP 502 responses. Its durable spool retried successfully and no accepted telemetry was lost, but the routed origin was not continuously available.

The corrected activation contract treats the candidate as a traffic handoff, not merely a preflight process. Tend validates and routes the imported Caddy handler to the candidate before changing the release pointer. It restarts and probes the fixed-address unit without public traffic, routes back only after that unit passes, and keeps the candidate healthy throughout the bounded activation window. Any failure restores the former pointer and exact handler bytes. If restoration itself cannot be completed, Tend leaves the proven candidate routed and running for explicit operator recovery instead of causing a known outage.

Regression tests cover successful handoff, restart failure, public-origin failure during the activation window, pointer and handler restoration, Caddy validation/reload boundaries, and candidate cleanup. Final production evidence must still repeat deploy, rollback, and reactivation with the exact release binary before this finding is treated as released.

F-10: archive modes under a hardened umask

Observation

The first two August 18 maintenance candidates stopped before activation. The release archives correctly recorded executable mode 0755, but extraction by root under umask 0077 left the installed application binary mode 0700. The unprivileged transient candidate could not execute it. Tend emitted failed candidate evidence, retained the prior release and route, and did not expose the failed binary to traffic.

Extraction now reapplies the already validated archive mode after file content is closed. A Linux regression test sets umask 0077, extracts the release, and requires the installed binary to remain executable by the service identity. The successful maintenance campaign used that corrected Tend binary.

F-12: retained-candidate reconciliation

Observation

An Observatory activation failure intentionally left its proven handoff candidate routed and running for operator recovery. A later push transferred and validated a new content-addressed artifact, then stopped before candidate startup because systemd still had Tend's fixed candidate unit loaded:

Unit observatory-tend-candidate.service was already loaded or has a fragment file.

The artifact, application, and host were valid. The collision was Tend-owned residue from its own safe fallback behavior. Recovery required starting the same stateless fallback under an independently named hardened unit, proving it, routing Caddy to it, and only then stopping the old candidate and confirming its port and unit name were free. No accepted application state was lost, but the operator had to understand Tend's internal unit convention.

Implemented foundation

• Transient candidate units are named by service plus the first 12 hexadecimal characters of a fresh 128-bit operation identity.
• Conventional deployment state remains schema 1 so the retained v0.1 binary can still read active and previous release identities. Singleton operations write a separate strict, adjacent candidate-lease file containing the operation, release, unit, address, and start time.
• tend reconcile --json reports persisted state, installed/candidate unit activity, release pointers, and whether the exact handler file matches the installed or candidate upstream. It explicitly distinguishes handler file bytes from Caddy's loaded runtime configuration and performs no mutation.
• A retained forward-activation or rollback candidate lease survives an incomplete recovery instead of being erased by a generic failed-attempt path. Older binaries ignore the additive lease file, so operators must reconcile before downgrading or attempting another activation.

Remaining options to assess

• Extend the candidate lease with separately observed health and loaded-route evidence rather than inferring either from a successful file write.
• Add an explicit tend resume operation that distinguishes active, routed, rollback, abandoned, and unknown candidates without mutating by default.
• Permit automatic cleanup only after Tend proves that Caddy, current/previous pointers, and the installed unit do not reference the candidate and that a healthy route remains.
• Make a repeated activation of the same approved digest idempotently resume the recorded operation instead of restaging or colliding with itself.

Never resolve the collision by blindly stopping the loaded unit. A retained candidate may be the only healthy route after a failed singleton activation.

F-13: public-origin topology and release identity

Observation

During the same recovery, the public DNS record had already moved to a new edge while the old host's Tend policy still named the canonical HTTPS origin as its post-activation smoke target. A request from the old host would therefore test a different deployment. It could fail even when the old host was healthy, or pass against a matching marker served by the new host. Neither result proves the release that Tend just activated.

Options to assess

• Separate a local routed-origin probe—Caddy with the canonical Host/SNI on the deployment host—from an external DNS-origin observation.
• Bind local success to a non-secret application release identity such as the approved artifact digest, commit, and version, not only a human page marker.
• Record the expected public edge identity or resolved address set at approval time. If it changes, classify the result as topology drift and require an explicit migration decision instead of reporting an application failure.
• Allow independent external observers as additional evidence, but never let a response from an unidentified deployment authorize activation.

Public reachability remains valuable; it must be evidence about the intended deployment rather than merely evidence that the hostname answered.

F-14: application data-path and preflight fidelity

Observation

An exact, checksummed Observatory data copy was mounted at a different absolute path in a container. Raw objects and durable SQLite databases were byte identical, but the application catalogue intentionally compared stored segment paths with filesystem-derived identities and failed closed. Preserving the original in-container data path made the same Preview 15 binary ready in four seconds with zero restarts and bounded memory.

This was an application portability constraint, not a reason for Tend to rewrite database state. It also demonstrates the limit of a binary-only stateless candidate: executable health cannot prove compatibility with the real configuration, credentials, mounts, or durable data.

SQLite -wal, -shm, and process-lock files also changed during ordinary open/close behavior while raw segments and durable databases did not. Migration evidence should distinguish durable application truth from ephemeral runtime coordination files.

Options to assess

• Support a fixed, bounded application-owned preflight argument vector with an explicit non-mutating contract, timeout, output limit, and no shell.
• Record the exact non-secret configuration digest, data-root identity, mount identity, and checks performed alongside binary candidate evidence.
• Let applications define their durable migration evidence set; Tend should transport and report those digests but must not infer database semantics or edit state.
• Require rollback compatibility to be assessed against the same configuration and data identities before an older binary is called safe.

F-15: facilitator rather than owner

Tend's strict boundaries are valuable where authority changes hands: source provenance, artifact digest approval, restricted transport, secret references, path validation, traffic movement, and rollback. Strictness becomes harmful when Tend's internal names or stale metadata are treated as application requirements that an authorized maintainer must reverse-engineer.

The target model is less invasive:

• systemd units, Caddy handlers, environment files, release directories, and application checks remain conventional and independently operable;
• Tend records an append-only migration/deployment journal and derives an observed state before proposing mutation;
• schema migrations preserve old records and explain compatibility rather than silently rejecting safe, recognizable state;
• check, status, and reconcile show facts and proposed repairs without mutation; approval is required for artifact selection and traffic movement, not for Tend implementation trivia;
• installation and adoption are explicit, while uninstalling Tend leaves an understandable, runnable service and complete evidence trail.

This does not relax hostile-input defenses. It moves strictness to the trust boundary and makes recovery humane for the authenticated operator.

F-16: ambiguous container-network identity

Observation

During the Observatory Preview 16 activation, the retained stateless handoff container and the live Compose service both answered to the Docker network alias observatory. A Caddy target such as observatory:8093 could therefore resolve to either release. Removing or restarting one container could also change which release received traffic. The application and both containers were healthy; the ambiguity existed solely in deployment identity.

The activation used a separately validated Caddy fragment pinned to the exact candidate address, proved its release marker before traffic moved, replaced the live service, and then restored the reviewed application handler. No ambiguous alias was used for the handoff.

Options to assess

• Give every candidate an operation-scoped container name, network alias, and lease that bind directly to its artifact digest and expected address.
• Resolve and inspect a proposed upstream immediately before Caddy validation; reject zero, multiple, or identity-mismatched targets.
• Prefer a dedicated candidate network or an address supplied by the container runtime over a stable alias shared with the live service.
• Record the exact routed target and release proof in activation state, then verify that the restored production handler identifies the active release.

Do not treat a healthy response from an ambiguous service name as release proof. Availability and deployment identity are separate properties.

F-17: CI artifact publication is not artifact availability

Observation

The trusted Observatory release-candidate workflow completed its two builds, byte comparison, checksums, manifest, and SBOM, and its upload action reported success. The forge artifact API subsequently returned no artifact for that run. The deployment client therefore had no independently discoverable object to download and inspect. Guessing a web-interface route or treating a green upload step as possession of the bytes would have weakened the build-approve- deploy boundary.

The release was instead reproduced twice from a fresh, clean clone of the exact pushed commit with the pinned toolchain. The packages compared byte-for-byte, their manifest, checksums, SBOM, version, commit, and tree were verified, and the approved digest was recorded. This preserved release identity, but it is an operator workaround rather than the desired CI handoff.

Options to assess

• Require the publisher to return a versioned receipt containing forge, run, artifact identifier, size, digest, retention, and retrieval endpoint.
• Add a separate read-only verification step that downloads the artifact using the same interface available to the approval client and reruns tend inspect before the workflow is considered publish-complete.
• Support a content-addressed, append-only artifact store whose object name is the approved SHA-256 and whose credentials remain separate from production.
• Let tend push accept only a locally present artifact plus an optional verified publication receipt; never allow a successful CI status alone to select bytes for deployment.

Artifact availability must be proven from the consumer side. A successful upload log is evidence of an attempted publication, not evidence that the approved bytes can be recovered.

The same failure repeated for Tend release-candidate run 369: the pinned job successfully built identical archives, recorded digest 2c225e0b9dd0be2d36fda62ab8d0b52cd9b28f9336c60dfb5edf3e715f450f95, and finalized a 3.18 MB upload, while both repository-wide and run-scoped REST artifact listings returned no objects. An independent clean build reproduced the exact digest before dogfooding. This confirms the problem is a reusable forge-to-approval gap rather than an Observatory-specific packaging defect.

F-18: Docker Compose is not a systemd singleton

Observation

Observatory Preview 18 was packaged reproducibly and its exact scratch image passed a constrained loopback candidate check. The production service on cliff-mads is a Docker Compose singleton, while Tend's supported strategies are systemd/Caddy blue-green and systemd singleton-candidate. Tend was therefore not used to pretend that an unsupported runtime had received a complete Tend activation proof.

The manual Compose replacement retained the former image and online database backups and completed health, readiness, route, projection, agent, and log checks. An independent public observer nevertheless recorded 200, then 502/503, then 200 across an approximately 31.6-second replacement and startup window. No application restart loop or accepted-data loss was observed, but this is not continuous delivery.

Preview.19 strengthened this finding. The exact candidate passed direct health and readiness, but an activation script treated the first transient routed 503 as final and attempted rollback before the proxy path had settled. The previous binary then correctly rejected the newly migrated control schema and entered a fail-closed restart loop. The operator recovered with the already proven forward candidate while the agent remained paused. This was a bounded, observable failure with intact backups, but it demonstrates that proxy settling and data-schema rollback are different gates and cannot share one generic failure branch.

Options to assess

• Add a distinct, versioned Docker Compose strategy rather than arbitrary command hooks or hidden container behavior inside existing strategies.
• Allocate an operation-scoped candidate container and network identity that cannot collide with the production alias.
• Validate the exact image digest, non-secret Compose/configuration digest, mounts, durable data identity, and application-owned read-only preflight before moving traffic.
• Route Caddy to the proven candidate, replace or promote the production service while the candidate remains healthy, then restore the reviewed live handler only after the final container proves its release identity.
• Restore the prior handler and retained image automatically when any activation-window probe fails; keep data rollback an explicit application-owned decision.

This strategy must remain optional. Tend should facilitate a conventional Compose service without requiring Tend-only container labels, aliases, or state for ordinary operator recovery.

F-19: binary rollback is not data rollback

Observation

Observatory Preview.19 migrated its live control database from the schema understood by Preview.18 to schema 11 before activation. When a premature public-origin failure triggered binary rollback, Preview.18 rejected the new schema rather than interpreting unknown state. That fail-closed behavior was correct, but Tend-like orchestration cannot infer from two binary identities whether their durable schemas are mutually readable.

The exact pre-migration control and projection databases had already been copied, integrity-checked, mode-restricted, and SHA-256 verified. The new candidate had also passed the migration on an isolated copy. Because the migration was forward-valid and the old agent was paused, completing the Preview.19 activation preserved more verified state than restoring the backup.

Options to assess

• Let an application declare a versioned, bounded compatibility statement for current-to-candidate and candidate-to-previous data access.
• Treat migration completion as a journalled phase after which automatic binary rollback is permitted only when backward readability was explicitly proven.
• Support an application-owned, separately approved data-restoration plan; never infer one from release pointers or run an arbitrary rollback hook.
• Keep verified backups and the proven forward candidate until the migration soak closes, even when the previous binary remains retained.

F-20: stateful preflight needs its real resource envelope

Observation

The Preview.19 stateless candidate and ordinary tests passed, but building a presence-only SQLite index over a copied production projection failed with database or disk is full under the service's 64 MiB /tmp tmpfs. The exact same candidate and data passed with a 512 MiB tmpfs and drained all pending raw segments. The projection database was healthy; the one-time index sort needed more bounded scratch space than steady-state operation.

Options to assess

• Record the non-secret service configuration digest and resource envelope beside the artifact digest.
• Permit a strict application-owned preflight command selected from a reviewed schema, never an arbitrary shell string.
• Exercise migrations against a copied or snapshot-backed production data set under the candidate's exact CPU, memory, temporary-storage, filesystem, and credential boundaries.
• Report resource exhaustion distinctly from corrupt data, failed health, or incompatible schema so recovery guidance remains accurate.

Prioritization

The recommended implementation order is:

1. operation-scoped candidate leases plus explicit resume/repair built on the new read-only reconciliation report;
2. migration compatibility and stateful-resource preflight contracts;
3. a bounded Docker Compose strategy with operation-scoped container/network identities and unambiguous upstream resolution and release proof;
4. release-bound local routed-origin proof separated from external DNS proof;
5. a bounded application preflight and configuration/data identity record;
6. a consumer-verifiable, digest-bound CI artifact publication receipt;
7. final production proof of the singleton traffic handoff;
8. a checksum-verified resolver and network-disabled package contract;
9. a provenance-preserving linked-worktree support decision after the new early diagnostic;
10. a standardized CI artifact/approval contract building on tend inspect;
11. exact release plus configuration identity in status and state;
12. restricted transfer and receive with host policy;
13. explicit adoption for existing services;
14. managed configuration only after its restoration and failure-injection model is as strong as binary activation.

Friction entries should be updated with the implementing version, tests, and dogfood evidence when resolved. Resolved entries remain in this ledger so the reason for the security boundary is not lost.