tend/docs/DOGFOOD_FRICTION.md

# Dogfood friction ledger

This document records friction observed through August 18, 2026 while using
Tend for real maintenance releases. Friction is evidence about the product: it
should become either a clearer contract, safer automation, or an explicit
non-goal. It must not become application-specific shell lore.

The findings below are maintainer observations, not implemented promises. Tend
continues to fail closed while an option is being designed. In particular,
there is no `--skip-remote` packaging escape hatch, no arbitrary deployment
hook, and no relaxation of the host-wide activation lock.

## Design standard

A Tend workflow should make the secure path the short, documented path:

- source, artifact, approval, configuration, and active-release identities are
  explicit and inspectable;
- credentials are narrowly scoped, briefly available, and absent from
  artifacts and logs;
- application validation is declarative, bounded, and cannot become a shell;
- first installation, maintenance, activation, rollback, and pruning are
  distinct operations;
- every mutation has a recorded prior state and a tested restoration path;
- unrelated services may build and prepare candidates concurrently, while the
  shared Caddy activation boundary remains serialized.

## Observed findings

| ID | Status | Finding | Current safe behavior | Candidate direction |
| --- | --- | --- | --- | --- |
| F-01 | Workflow mitigation | CI packaging needs proof that the exact commit was pushed. | Packaging verifies the configured remote and fails if Git cannot authenticate. The trusted workflow retains only its job-scoped read-only checkout credential. | Make the proof credential or an authenticated source attestation an explicit Tend input and evidence boundary. |
| F-02 | Open | Application-owned unit and configuration changes are outside the binary activation transaction. | Operators stage, validate, back up, and restore those files separately. | Add an allowlisted configuration transaction or record and validate exact configuration digests. |
| F-03 | Open | A singleton candidate may not reproduce the installed unit's arguments and application configuration. | It receives the production environment file and an isolated listen override; activation still fails closed. | Add a bounded application preflight and explicit, validated candidate invocation. |
| F-04 | Partially resolved | Artifact production, review, approval, transfer, and activation require several identity checks. | v0.2 standardizes the evidence bundle and restricted transfer; the operator still repeats the approved digest intentionally. | Add offline inspection and approval ergonomics without combining build authority and production authority. |
| F-05 | Partially resolved | Tend maintenance assumes an existing adopted service and current release. | `check-server` validates a prepared host, while first installation remains a separately reviewed operator procedure. | Define an explicit `install` or `adopt` transaction rather than silently treating bootstrap as maintenance. |
| F-06 | Partially resolved | Binary rollback can be unsafe when service configuration has changed incompatibly. | v0.2 records desired, candidate, active, previous, and last-attempt releases, but external configuration compatibility is operator-owned. | Bind non-secret configuration identities and preflight results to release state. |
| F-07 | Open | `GOPROXY=off` does not prove that every module metadata lookup is available locally. | Packaging stops before artifact creation when Go cannot resolve the complete pinned module graph. | Separate checksum-verified dependency resolution from a network-disabled, cache-completeness-checked build stage. |
| F-08 | Open | Packaging an otherwise clean pushed commit from a linked Git worktree fails Go's required VCS-status stamp. | Tend stops before artifact creation; package the same exact commit from a clean standalone clone. | Detect linked worktrees during `check`/`package`, explain the supported source shape, and assess a provenance-preserving worktree build that does not weaken `-buildvcs=true`. |
| F-09 | Resolved in Preview 2 | Restarting a fixed-address singleton briefly exposed Caddy to an unavailable upstream. | Tend now routes the imported handler to the candidate, restarts and proves the installed unit behind that handoff, then routes back while the candidate remains healthy through the activation window. | Preserve the production regression campaign and failure-injection matrix. |
| F-10 | Resolved in Preview 2 | A hardened root umask could make an extracted application binary executable only by root. | Tend reapplies every validated archive mode explicitly; extraction is tested under umask `0077`. | Preserve the mode and non-root candidate tests. |
| F-11 | Evidence practice | A single external observer can report common-mode client or network errors as apparent multi-service downtime. | Production campaigns retain server state and use an independent observer before classifying a continuity failure. | Standardize multi-vantage continuity evidence without making an observability dependency part of deployment authority. |

## F-01: pushed-commit proof in CI

### Observation

During an Observatory release-candidate run, verification intentionally checked
out source without persisted credentials. Tend later attempted to prove that
`HEAD` existed on the configured private Gitea branch and Git could not
authenticate. Packaging stopped before producing an artifact. This was the
correct fail-closed outcome, but the credential lifecycle was not obvious from
the workflow contract.

The trusted workflow uses a job-scoped read-only checkout credential so Tend
can perform the independent remote proof. It is not a long-lived repository or
deployment credential, and it is removed by checkout cleanup at the end of the
job. Production credentials remain unavailable to the build job.

### Options to assess

1. Support an explicit read-only Git credential file or credential helper for
   package-time proof. Never accept a token in arguments, configuration,
   manifests, artifacts, or logs.
2. Accept an authenticated CI source-attestation document binding repository,
   branch, commit, tree, workflow identity, and event identity. Define which
   CI issuers are trusted and preserve the attestation with release evidence.
3. Accept a signed, pre-verified source bundle whose identity and policy can be
   checked without network access.

Do not add a generic skip flag. An unavailable proof must remain a packaging
failure unless an equally strong proof mode was selected explicitly.

## F-02: application configuration is not binary activation

### Observation

Tend can transact an immutable binary, release pointers, a systemd restart, a
validated Caddy handler, health checks, and rollback state. It does not
currently transact application configuration, systemd unit changes, credential
bindings, or server-local secret files. A first deployment that changes these
files therefore needs a separate backup, validation, installation, and
restoration procedure.

This boundary is safe but easy to overlook: restoring the previous binary does
not restore an incompatible unit or application configuration.

### Options to assess

1. Add strict managed-file entries with exact source and destination paths,
   content digest, owner, group, mode, and validation type. Permit only regular
   files beneath configured roots; reject symlinks and unknown destinations.
2. Add a separate `tend configure` transaction that backs up, atomically
   replaces, validates, and restores supported systemd/Caddy/application files.
3. Keep configuration externally managed, but require Tend to record desired
   and active configuration digests and prove candidate/rollback compatibility.

None of these options should permit arbitrary shell commands. Secret values
remain referenced server-side and must never enter release artifacts or state.

## F-03: candidate fidelity

### Observation

A hardened singleton candidate receives the configured production environment
file and Tend's isolated listen-address override. It does not currently model
the installed systemd unit's complete argument vector. An application whose
configuration path is supplied by unit arguments may therefore start a
candidate with defaults instead of the intended production configuration.
That preserves live-state isolation, but it may prove only executable startup.
A missing credential, incompatible configuration field, filesystem permission,
or data migration requirement can then surface at activation time.

### Options to assess

- Define a fixed application preflight command or protocol that validates the
  production configuration, credentials, permissions, and data compatibility
  without binding the live port or mutating live state.
- Add a strict candidate argument vector to schema 2. Validate each argument,
  reject secret values and paths outside the application contract, and pass it
  directly to systemd without a shell.
- Permit an allowlisted `check` argument vector, never a shell string, with
  bounded time/output and an explicitly non-mutating application contract.
- Record which checks ran against the binary alone and which ran against the
  actual production configuration so the evidence cannot overstate coverage.

## F-04: artifact review and approval ergonomics

### Observation

The secure flow deliberately separates build, review, digest approval,
transfer, candidate validation, and activation. v0.2 now produces a consistent
archive, manifest, SBOM, and checksum set, then transfers one exact artifact
through its restricted receiver. In practice, operators still need a
predictable way to discover that CI artifact, inspect it, approve exactly one
digest, and retain the review evidence. Without a first-class review path,
correct manual steps are easy to reconstruct differently for each application.

### Options to assess

- Add `tend inspect` for offline, non-mutating verification and a concise human
  and JSON summary of source, build, dependency, and archive identities.
- Add an approval record that binds the artifact digest, service, target,
  approver, and expiry without containing a credential.
- Standardize one Gitea artifact layout and documented download-to-activation
  workflow. Keep production credentials unavailable to verification jobs.

Approval must remain explicit; better ergonomics must not turn a successful
build into an automatic production mutation.

## F-05: first installation and adoption

### Observation

The maintenance workflow expects an installed service, valid configuration,
and a current release pointer. `check-server` validates the restricted receiver
policy and prepared service boundary, but it does not create them. Observatory
bootstrap therefore required operator-managed service account, directories,
credentials, unit, configuration, and an initial current release before Tend
could own later maintenance safely.

### Options to assess

- `tend install`: a deliberately broader, separately approved transaction with
  a strict schema and complete rollback of every supported created object.
- `tend adopt`: validate an existing service and release, copy or identify its
  immutable artifact, establish state, and refuse ambiguous ownership.
- Keep bootstrap out of Tend, but ship a versioned acceptance checklist and a
  machine-readable `check-server` result that maintenance can require.

Installation and adoption must not be inferred from a missing state file.

## F-06: release and configuration identity

### Observation

Content-addressed releases and v0.2 state make desired, candidate, active,
previous, and last-attempt binary identities clear. A live service is still a
combination of its binary, application configuration, credential bindings,
unit, routing fragment, and sometimes data schema. Tend cannot yet fully
explain that combined identity or determine whether a retained binary is
compatible with the current external configuration.

### Options to assess

- Extend the existing desired, candidate, active, previous, and last-attempt
  release state with non-secret configuration and unit digests.
- Require rollback compatibility declarations or read-only application
  preflight before changing traffic.
- Expose the identities and last validation results through
  `tend status --json` for deployment evidence and future Observatory
  ingestion.

Configuration records contain digests and approved metadata only—not secret
contents.

## F-07: offline module-cache completeness

### Observation

An exact Observatory package attempt used `GOPROXY=off` and a previously used
module cache. Source archives for the application dependencies were present,
but Go still needed several module metadata records while Tend enumerated the
complete build graph. Go refused the lookup and Tend stopped before building
or writing an artifact. Re-enabling the checksum-verified public proxy supplied
the missing metadata; the resulting package was byte-identical to the trusted
CI candidate.

This is safe failure, but `GOPROXY=off` alone is not evidence of a hermetic
build. A cache can be partially populated even when ordinary builds happen to
succeed.

### Options to assess

- Add a resolver stage that runs with the pinned toolchain, proxy, and checksum
  database, emits the complete module inventory, and materializes a bounded,
  read-only cache for the builder.
- Run Tend's package stage with networking disabled and require every module,
  checksum, source archive, and metadata record to come from that reviewed
  cache.
- Add a non-mutating cache-completeness check that reports missing module
  identities before the expensive double build.
- Consider a strictly verified vendored-source mode where repository size and
  update review are acceptable; do not silently change dependency modes.

Do not treat `GONOSUMDB`, `GOPRIVATE`, or a proxy bypass as an offline-build
solution. They alter verification or routing policy rather than proving cache
completeness.

## F-08: linked-worktree VCS stamping

### Observation

An exact clean, pushed Observatory main commit passed verification and public
snapshot isolation from a detached linked Git worktree. Tend accepted its
source and remote provenance, then Go 1.26.6 stopped both `tend package` and an
equivalent direct build at the required `-buildvcs=true` step with `error
obtaining VCS status: exit status 128`. Ordinary Git status and commit queries
from the same worktree succeeded.

A fresh standalone SSH clone of the identical commit packaged successfully
twice. Both archives were byte-identical and carried the expected commit, Go
version, VCS settings, checksums, manifest, and SBOM. This isolates the failure
to the linked-worktree build shape rather than the application source or module
graph.

The failure is safe: Tend did not create a partial artifact and must not switch
to `-buildvcs=false`, because the package gate independently verifies the
embedded VCS revision and clean state. Until the interaction is resolved, use
a clean standalone clone for release packaging.

### Options to assess

- Detect a `.git` indirection file during `tend check` and fail before the
  expensive double build with a precise standalone-clone instruction.
- Reproduce the interaction in a package integration test against the minimum
  supported Go release and determine whether it is a Go toolchain limitation
  or an invocation/environment defect.
- If linked worktrees can be supported, require the resulting build record to
  carry the exact expected `vcs.revision` and `vcs.modified=false`; do not
  synthesize those settings or disable VCS stamping.
- Consider an explicit, signed source-bundle input as part of the broader
  source-attestation design. It must remain at least as strong as current
  pushed-commit proof.

## F-09: singleton traffic continuity

### Observation

During the August 18 Observatory maintenance exercise, the transient candidate
passed health, readiness, and content checks. Tend then stopped that candidate,
changed the singleton pointer, and restarted the installed fixed-address unit.
Caddy still targeted the fixed address during that restart, so the Observatory
agent observed a small number of transient HTTP `502` responses. Its durable
spool retried successfully and no accepted telemetry was lost, but the routed
origin was not continuously available.

The corrected activation contract treats the candidate as a traffic handoff,
not merely a preflight process. Tend validates and routes the imported Caddy
handler to the candidate before changing the release pointer. It restarts and
probes the fixed-address unit without public traffic, routes back only after
that unit passes, and keeps the candidate healthy throughout the bounded
activation window. Any failure restores the former pointer and exact handler
bytes. If restoration itself cannot be completed, Tend leaves the proven
candidate routed and running for explicit operator recovery instead of causing
a known outage.

Regression tests cover successful handoff, restart failure, public-origin
failure during the activation window, pointer and handler restoration, Caddy
validation/reload boundaries, and candidate cleanup. Final production evidence
must still repeat deploy, rollback, and reactivation with the exact release
binary before this finding is treated as released.

## F-10: archive modes under a hardened umask

### Observation

The first two August 18 maintenance candidates stopped before activation. The
release archives correctly recorded executable mode `0755`, but extraction by
root under umask `0077` left the installed application binary mode `0700`.
The unprivileged transient candidate could not execute it. Tend emitted failed
candidate evidence, retained the prior release and route, and did not expose
the failed binary to traffic.

Extraction now reapplies the already validated archive mode after file content
is closed. A Linux regression test sets umask `0077`, extracts the release, and
requires the installed binary to remain executable by the service identity.
The successful maintenance campaign used that corrected Tend binary.

## Prioritization

The recommended implementation order is:

1. final production proof of the singleton traffic handoff;
2. a checksum-verified resolver and network-disabled package contract;
3. an early linked-worktree diagnostic and a provenance-preserving support
   decision;
4. `tend inspect` and a standardized CI artifact/approval contract;
5. exact release plus configuration identity in status and state;
6. a bounded application preflight contract;
7. restricted transfer and receive with host policy;
8. explicit adoption for existing services;
9. managed configuration only after its restoration and failure-injection
   model is as strong as binary activation.

Friction entries should be updated with the implementing version, tests, and
dogfood evidence when resolved. Resolved entries remain in this ledger so the
reason for the security boundary is not lost.