Author SHA1 Message Date
gamertan b2cc4482f6 fix: prepare Tend preview 2
Publish the reviewed version identity fix and mark preview 1 withdrawn.\n\nPrivate-Source-Commit: 9907afdfa18099ac488fe4011291ea454036435a\nPrivate-Source-Tree: 3a5e666dd6a0c69a07f7804612c8e3dc3474966a\nHimesan-Output-Permission: v1.0\nAI-Assistance: OpenAI Codex assisted implementation and review.

Signed-off-by: Cole Speelman <crspeelman@gmail.com>
2026-08-14 14:19:01 -04:00
gamertan b68fa2487d feat: publish Gamertan Tend preview source
Sanitized root snapshot from private source commit a72903c63e1753f9e6ffbf40453c0830bdfc05c5 and tree 295641e67eef5979da76746d8ae271249568263e. Private development history and workflows are excluded by the exact allowlist.

AI-assisted: OpenAI Codex helped implement, test, and audit this preview.
Signed-off-by: Cole Speelman <crspeelman@gmail.com>
2026-08-14 14:01:02 -04:00
65 changed files with 244 additions and 4622 deletions
-2
View File
@@ -1,5 +1,3 @@
/bin/ /bin/
/dist/ /dist/
*.tmp *.tmp
.env.local
*.env.local
+1 -1
View File
@@ -1 +1 @@
{"schema_version":1,"source_commit":"07c1655921f21ee5e4fc4d85639d199e8867b17d","source_tree":"0a27c7c0d2fb8954e326044d06826cf7e1a37818","source_date_epoch":1787103648,"file_count":81} {"schema_version":1,"source_commit":"a72903c63e1753f9e6ffbf40453c0830bdfc05c5","source_tree":"295641e67eef5979da76746d8ae271249568263e","source_date_epoch":1786730082,"file_count":44}
+1 -1
View File
@@ -1 +1 @@
a38ba7894dede461d7aec3f1b07c5c6f65728f2971b973fc81c469c32667d38e PUBLIC-SNAPSHOT.json b29d919cefddb06e0c47b453ffd55438944fc92da5732dbb6f98cb509a2c371e PUBLIC-SNAPSHOT.json
+12 -83
View File
@@ -1,23 +1,11 @@
# Gamertan Tend # Gamertan Tend
## Final public CLI preview
`v0.2.0-preview.2` is the final public Tend CLI preview. This repository and
its published AGPL releases remain available as an archived record, but it is
not accepting new feature development. Private research continues around a
networked management experience; that work is not mirrored or published here
and carries no public release commitment.
Existing licences and release artifacts are unchanged. Do not interpret this
notice as withdrawing rights already granted by the AGPL-3.0-only and 0BSD
files in this repository.
Tend is an opinionated release and deployment tool for small Go services on Tend is an opinionated release and deployment tool for small Go services on
Linux, systemd, and Caddy. It packages a clean pushed commit, records exact Linux, systemd, and Caddy. It packages a clean pushed commit, records exact
build provenance, activates a health-checked candidate, and keeps rollback build provenance, activates a health-checked candidate, and keeps rollback
state explicit. state explicit.
The v0.2 development line supports two quiet deployment shapes: The first preview deliberately supports two quiet deployment shapes:
- Caddy-switched blue/green services. - Caddy-switched blue/green services.
- A singleton service with an isolated transient candidate check. - A singleton service with an isolated transient candidate check.
@@ -29,80 +17,21 @@ shell hooks. Application-specific data activation remains application-specific.
```text ```text
tend check --config tend.json tend check --config tend.json
tend package --config tend.json --version v0.2.0-preview.2 --out dist tend package --config tend.json --version v0.1.0-preview.2 --out dist
tend inspect --config tend.json --artifact FILE --sha256 HEX --approve-sha256 HEX tend deploy --config /etc/example/tend.json --artifact FILE --sha256 HEX --approve-sha256 HEX
tend push --target tend-deploy@host --known-hosts FILE --service NAME --artifact FILE --sha256 HEX --approve-sha256 HEX tend status --config /etc/example/tend.json
tend receive --policy /etc/tend/receive-policy.json tend rollback --config /etc/example/tend.json
tend check-server --policy /etc/tend/receive-policy.json tend prune --config /etc/example/tend.json --keep 3 [--apply]
tend deploy --config /etc/tend/services/example-site.json --artifact FILE --sha256 HEX --approve-sha256 HEX
tend status --config /etc/tend/services/example-site.json
tend reconcile --config /etc/tend/services/example-site.json --json
tend rollback --config /etc/tend/services/example-site.json
tend prune --config /etc/tend/services/example-site.json --keep 3 [--apply]
``` ```
`inspect` performs the complete archive, checksum, approval, manifest, SBOM, `deploy`, `rollback`, and applied pruning are target-host operations. Transfer
and binary-identity validation without staging or activating a release. credentials and production configuration stay outside source repositories.
`reconcile` is also read-only: it compares Tend's journal with conventional Dry-run and digest approval are intentional friction.
release symlinks, systemd units, the installed release identity, and the Caddy
handler. It reports drift and retained-candidate residue without silently
"repairing" a service.
`push` transfers one approved artifact through a pinned OpenSSH connection. A The first public preview has completed maintenance releases and explicit
forced, no-shell receiver maps the service name to one root-owned configuration; rollback/reactivation for both Gamertan and the Sandwich Hime website using one
it accepts no remote path, environment value, URL, or shell fragment. Production reviewed Tend candidate. See the dated
hosts receive binaries and evidence, never source or Go dependencies.
Schema 2 keeps all services under `/etc/tend/services/`, references a distinct
root-owned `0600` environment file for each service, and serializes activation
through `/run/lock/tend-deploy.lock`. Builds and transfers remain parallel;
only the short Caddy/service activation phase is host-wide. Tend is still a
single command, not a daemon.
Each service also keeps a bounded JSONL deployment-event stream and explicit
desired, candidate, active, previous, and last-attempt release identities. The
stream contains only fixed provenance and lifecycle fields; Observatory may
ingest it later, but an event-write failure never blocks deployment or rollback.
After Caddy reload, Tend repeatedly probes the configured canonical HTTPS
origins for the activation window. Blue/green deployments simultaneously keep
checking the previous slot, restoring the prior handler and inactive-slot state
if routed traffic or continuity fails.
Singleton activation and rollback candidates use an operation-scoped systemd
unit and persist a bounded lease containing the operation, release, unit,
address, and start time in a separate adjacent file. Conventional deployment
state remains schema 1 so a retained older binary can still read active and
previous release identities; operators must reconcile before downgrading while
a lease exists. `tend
reconcile --json` compares that lease with unit activity, release pointers, and
the exact Caddy handler file without changing any of them. It deliberately does
not stop or clear a retained candidate: that process may still be the only
healthy route after an interrupted recovery.
Dry-run and digest approval are intentional friction. See the
[schema-2 migration guide](docs/SCHEMA_V2_MIGRATION.md) and the
[two-service walkthrough](docs/WALKTHROUGH.md).
Run `./scripts/verify.sh` on Linux. That required release lane exercises tests,
the race detector, vet, deterministic builds, schema-2 examples, and the
dependency-free module graph. Tend supports Linux hosts with systemd and Caddy;
WSL may be used as a Linux development environment, but native Windows is not a
supported execution, deployment, or release-gate platform.
The v0.1 public preview and immutable `v0.2.0-preview.1` each completed
maintenance releases and explicit rollback/reactivation for both Gamertan and
the Sandwich Hime website using one reviewed candidate. See the dated
[dogfood evidence](docs/DOGFOOD_EVIDENCE.md) for exact scope and limitations. [dogfood evidence](docs/DOGFOOD_EVIDENCE.md) for exact scope and limitations.
The additive `v0.2.0-preview.2` release keeps that transport and activation
contract while adding bounded deployment-event JSONL, routed-origin continuity,
rollback annotations, and the operational findings recorded through real
dogfooding. Preview 1 remains unchanged.
Operational friction discovered while applying the same contract to new
services is tracked separately in the
[dogfood friction ledger](docs/DOGFOOD_FRICTION.md). The ledger preserves the
fail-closed behavior and records candidate product improvements instead of
normalizing application-specific deployment workarounds.
The canonical public repository begins with a sanitized root snapshot rather The canonical public repository begins with a sanitized root snapshot rather
than the private development history. than the private development history.
-20
View File
@@ -19,26 +19,6 @@ The release tag and attached candidate must be built from the final reviewed
source commit. Documentation-only changes after the recorded campaign require source commit. Documentation-only changes after the recorded campaign require
one final identical-candidate maintenance pass before tagging. one final identical-candidate maintenance pass before tagging.
`v0.2.0-preview.1` is an immutable, additive release line. It requires schema 2,
the restricted `push`/`receive` transport, root-owned environment-file
references, host-wide activation serialization, and HTTPS public-origin smoke.
Its exact released source remains unchanged.
`v0.2.0-preview.2` adds bounded deployment-event JSONL, routed-origin
activation continuity, rollback annotations, and the reviewed operational
friction record. It must preserve every Preview 1 security and release gate.
The exact same Preview 2 binary must deploy, roll back, and reactivate
Gamertan, the Sandwich Hime website, and Gamertan Observatory before the tag is
created. EQL is not part of this generic gate; its SQLite/catalog publication
needs a dedicated adapter rather than arbitrary hooks.
The operation-scoped lease compatibility candidate at private commit
`789976ce766fd457ca10762540135e4e0e74cfe3` has completed the Gamertan and
Sandwich Hime deployment, rollback, reactivation, and failure gates. It has not
completed the Observatory gate because the live service now uses Docker
Compose, outside the released strategy contract. Do not create the Preview 2
tag from this candidate until that explicit topology decision is resolved.
`v0.1.0-preview.1` is immutable but withdrawn: its source and module checksums `v0.1.0-preview.1` is immutable but withdrawn: its source and module checksums
are valid, while a fresh `go install` reports the development identity because are valid, while a fresh `go install` reports the development identity because
the CLI did not yet adopt the tagged module version from Go build information. the CLI did not yet adopt the tagged module version from Go build information.
+4 -7
View File
@@ -4,14 +4,11 @@ Report suspected vulnerabilities privately to `security@sandwichhime.com`.
Please include the affected Tend version, configuration shape, reproduction, Please include the affected Tend version, configuration shape, reproduction,
and expected impact. Do not include production credentials or private logs. and expected impact. Do not include production credentials or private logs.
Tend treats reviewed source, pinned toolchains, root-owned server policy and Tend treats source repositories, release artifacts, configuration, handwritten
configuration, handwritten Caddy templates, and operators as trusted. It treats Caddy templates, and operators as trusted. It treats artifact paths, archives,
requested service names, protocol frames, artifact paths and bytes, archives,
filesystem state, process output, HTTP responses, and deployment targets as filesystem state, process output, HTTP responses, and deployment targets as
adversarial inputs. It never evaluates configuration as shell code. adversarial inputs. It never evaluates configuration as shell code.
The preview is not a sandbox and does not make an untrusted repository safe to The preview is not a sandbox and does not make an untrusted repository safe to
build. Run `tend package` only for reviewed source. Production configuration, build. Run `tend package` only for reviewed source. Production configuration
receive policy, host keys, identities, and `0600` environment files stay outside must be root-owned and kept outside repositories.
repositories. A secret value appearing in a Tend report, artifact, state file,
or process argument is a security defect and should be reported.
+13 -156
View File
@@ -19,8 +19,6 @@ import (
"gamertan.com/tend/internal/deploy" "gamertan.com/tend/internal/deploy"
"gamertan.com/tend/internal/packager" "gamertan.com/tend/internal/packager"
"gamertan.com/tend/internal/process" "gamertan.com/tend/internal/process"
"gamertan.com/tend/internal/serverpolicy"
"gamertan.com/tend/internal/transport"
"gamertan.com/tend/internal/version" "gamertan.com/tend/internal/version"
) )
@@ -43,18 +41,8 @@ func run() error {
return packageCommand(ctx, os.Args[2:]) return packageCommand(ctx, os.Args[2:])
case "deploy": case "deploy":
return deployCommand(ctx, os.Args[2:]) return deployCommand(ctx, os.Args[2:])
case "inspect":
return inspectCommand(ctx, os.Args[2:])
case "push":
return pushCommand(ctx, os.Args[2:])
case "receive":
return receiveCommand(ctx, os.Args[2:])
case "check-server":
return checkServerCommand(os.Args[2:])
case "status": case "status":
return statusCommand(ctx, os.Args[2:]) return statusCommand(ctx, os.Args[2:])
case "reconcile":
return reconcileCommand(ctx, os.Args[2:])
case "rollback": case "rollback":
return rollbackCommand(ctx, os.Args[2:]) return rollbackCommand(ctx, os.Args[2:])
case "prune": case "prune":
@@ -66,7 +54,7 @@ func run() error {
} }
} }
func usage() error { func usage() error {
return errors.New("usage: tend <check|package|inspect|push|receive|check-server|deploy|status|reconcile|rollback|prune|version> [options]") return errors.New("usage: tend <check|package|deploy|status|rollback|prune|version> [options]")
} }
func checkCommand(args []string) error { func checkCommand(args []string) error {
@@ -135,7 +123,7 @@ func deployCommand(ctx context.Context, args []string) error {
return err return err
} }
if *activate { if *activate {
if err := requireMutationAuthority(absolute, cfg); err != nil { if err := requireMutationAuthority(absolute); err != nil {
return err return err
} }
} }
@@ -149,123 +137,6 @@ func deployCommand(ctx context.Context, args []string) error {
} }
return writeJSON(report) return writeJSON(report)
} }
func inspectCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("inspect", flag.ContinueOnError)
set.SetOutput(os.Stderr)
path := set.String("config", "", "target configuration")
artifact := set.String("artifact", "", "release archive")
sha := set.String("sha256", "", "expected artifact SHA-256")
approved := set.String("approve-sha256", "", "separately reviewed artifact SHA-256")
if err := set.Parse(args); err != nil {
return err
}
if set.NArg() != 0 {
return errors.New("inspect accepts no positional arguments")
}
cfg, _, err := loadConfig(*path)
if err != nil {
return err
}
artifactAbs, err := filepath.Abs(*artifact)
if err != nil {
return err
}
report, err := newManager().Deploy(ctx, cfg, deploy.Request{
Artifact: artifactAbs,
SHA256: *sha,
ApprovedSHA256: *approved,
Activate: false,
})
if err != nil {
return err
}
return writeJSON(report)
}
func pushCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("push", flag.ContinueOnError)
set.SetOutput(os.Stderr)
target := set.String("target", "", "dedicated tend-deploy user and host")
port := set.Int("port", 22, "pinned SSH port")
knownHosts := set.String("known-hosts", "", "absolute pinned known-hosts file")
identity := set.String("identity", "", "optional absolute private key")
service := set.String("service", "", "allowed service name")
artifact := set.String("artifact", "", "immutable release archive")
sha := set.String("sha256", "", "expected artifact SHA-256")
approved := set.String("approve-sha256", "", "separately reviewed artifact SHA-256")
activate := set.Bool("activate", false, "activate after server validation")
if err := set.Parse(args); err != nil {
return err
}
if set.NArg() != 0 {
return errors.New("push accepts no positional arguments")
}
artifactAbs, err := filepath.Abs(*artifact)
if err != nil {
return err
}
result, err := transport.Push(ctx, transport.ExecSSHRunner{}, transport.PushOptions{Target: *target, Port: *port, KnownHosts: *knownHosts, Identity: *identity, Service: *service, Artifact: artifactAbs, SHA256: *sha, ApprovedSHA256: *approved, Activate: *activate})
if err != nil {
return err
}
return writeJSON(result)
}
func receiveCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("receive", flag.ContinueOnError)
set.SetOutput(os.Stderr)
policyPath := set.String("policy", "/etc/tend/receive-policy.json", "root-owned receive policy")
if err := set.Parse(args); err != nil {
return err
}
if set.NArg() != 0 {
return errors.New("receive accepts no positional arguments")
}
if runtime.GOOS != "linux" || os.Geteuid() != 0 {
return errors.New("receive requires Linux root execution")
}
if os.Getenv("SSH_ORIGINAL_COMMAND") != transport.Protocol {
return errors.New("receive refused unexpected SSH command")
}
policy, err := serverpolicy.Load(*policyPath)
if err != nil {
return err
}
report, err := transport.Receive(ctx, os.Stdin, policy, newManager())
if err != nil {
return err
}
return writeJSON(report)
}
func checkServerCommand(args []string) error {
set := flag.NewFlagSet("check-server", flag.ContinueOnError)
set.SetOutput(os.Stderr)
policyPath := set.String("policy", "/etc/tend/receive-policy.json", "root-owned receive policy")
if err := set.Parse(args); err != nil {
return err
}
if set.NArg() != 0 {
return errors.New("check-server accepts no positional arguments")
}
if runtime.GOOS != "linux" || os.Geteuid() != 0 {
return errors.New("check-server requires Linux root execution")
}
policy, err := serverpolicy.Load(*policyPath)
if err != nil {
return err
}
services, err := policy.CheckFiles()
if err != nil {
return err
}
names := make([]string, 0, len(services))
for _, service := range services {
names = append(names, service.Name)
}
return writeJSON(map[string]any{"valid": true, "schema_version": policy.SchemaVersion, "services": names})
}
func statusCommand(ctx context.Context, args []string) error { func statusCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("status", flag.ContinueOnError) set := flag.NewFlagSet("status", flag.ContinueOnError)
set.SetOutput(os.Stderr) set.SetOutput(os.Stderr)
@@ -286,27 +157,6 @@ func statusCommand(ctx context.Context, args []string) error {
} }
return writeJSON(result) return writeJSON(result)
} }
func reconcileCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("reconcile", flag.ContinueOnError)
set.SetOutput(os.Stderr)
path := set.String("config", "", "absolute target configuration")
_ = set.Bool("json", false, "emit the reconciliation report as JSON")
if err := set.Parse(args); err != nil {
return err
}
if set.NArg() != 0 {
return errors.New("reconcile accepts no positional arguments")
}
cfg, _, err := loadConfig(*path)
if err != nil {
return err
}
report, err := newManager().Reconcile(ctx, cfg)
if err != nil {
return err
}
return writeJSON(report)
}
func rollbackCommand(ctx context.Context, args []string) error { func rollbackCommand(ctx context.Context, args []string) error {
set := flag.NewFlagSet("rollback", flag.ContinueOnError) set := flag.NewFlagSet("rollback", flag.ContinueOnError)
set.SetOutput(os.Stderr) set.SetOutput(os.Stderr)
@@ -325,7 +175,7 @@ func rollbackCommand(ctx context.Context, args []string) error {
if !*activate { if !*activate {
return writeJSON(map[string]any{"validated": true, "mutation": "none", "message": "repeat with --activate to switch to the recorded previous release"}) return writeJSON(map[string]any{"validated": true, "mutation": "none", "message": "repeat with --activate to switch to the recorded previous release"})
} }
if err := requireMutationAuthority(absolute, cfg); err != nil { if err := requireMutationAuthority(absolute); err != nil {
return err return err
} }
record, err := newManager().Rollback(ctx, cfg) record, err := newManager().Rollback(ctx, cfg)
@@ -351,7 +201,7 @@ func pruneCommand(args []string) error {
return err return err
} }
if *apply { if *apply {
if err := requireMutationAuthority(absolute, cfg); err != nil { if err := requireMutationAuthority(absolute); err != nil {
return err return err
} }
} }
@@ -375,14 +225,21 @@ func loadConfig(path string) (config.Config, string, error) {
func newManager() deploy.Manager { func newManager() deploy.Manager {
return deploy.NewManager(deploy.SystemOperator{Runner: process.ExecRunner{}, Timeout: 5 * time.Second}) return deploy.NewManager(deploy.SystemOperator{Runner: process.ExecRunner{}, Timeout: 5 * time.Second})
} }
func requireMutationAuthority(configPath string, cfg config.Config) error { func requireMutationAuthority(configPath string) error {
if runtime.GOOS != "linux" { if runtime.GOOS != "linux" {
return errors.New("deployment mutations require Linux") return errors.New("deployment mutations require Linux")
} }
if os.Geteuid() != 0 { if os.Geteuid() != 0 {
return errors.New("deployment mutations require root") return errors.New("deployment mutations require root")
} }
return serverpolicy.CheckConfig(configPath, cfg) info, err := os.Stat(configPath)
if err != nil {
return err
}
if info.Mode().Perm()&0o022 != 0 {
return errors.New("production configuration must not be group- or world-writable")
}
return nil
} }
func writeJSON(value any) error { func writeJSON(value any) error {
encoder := json.NewEncoder(os.Stdout) encoder := json.NewEncoder(os.Stdout)
+12 -60
View File
@@ -17,26 +17,12 @@ requires byte-identical output. The resulting archive contains only:
The external archive digest is the release identity used by deployment. The external archive digest is the release identity used by deployment.
## Transport boundary
`tend push` sends one bounded protocol frame to a dedicated account through a
pinned OpenSSH host key. It uses an argument vector, disables config files,
forwarding, PTYs, local commands, and proxy commands, and requests exactly
`tend-receive-v1`. The account's forced command invokes only the root-owned
receiver. The receiver's root-owned policy maps an allowlisted service name to
one exact configuration path and size ceiling.
The target receives no source, repository credential, Go cache, dependency, or
arbitrary command. The artifact digest must equal both the produced digest and
the separately supplied approved digest before it is staged.
## Target-host boundary ## Target-host boundary
Schema-2 configurations live below `/etc/tend/services/`. Secrets live only in `tend deploy` runs on the Linux target. Transfer and SSH authentication stay
separate `/etc/tend/environment/*.env` files that are root-owned, non-symlink, outside Tend v0.1; application-local transfer commands move the artifact and
and mode `0600`. Configurations contain the path, never the values. Candidate the identical Tend binary to a root-controlled staging directory. Tend then
and installed units read the same file; Tend overrides only the candidate's requires the expected digest to be repeated as a separately approved digest.
loopback listen address.
Dry-run validation extracts into a temporary directory and does not touch the Dry-run validation extracts into a temporary directory and does not touch the
release tree. Activation acquires the shared lock, installs a content-addressed release tree. Activation acquires the shared lock, installs a content-addressed
@@ -49,46 +35,12 @@ Blue/green mode points the inactive slot at the new release, restarts and
probes it, atomically replaces one imported Caddy handler, validates the full probes it, atomically replaces one imported Caddy handler, validates the full
Caddy configuration, reloads Caddy, and records the prior active slot. Caddy configuration, reloads Caddy, and records the prior active slot.
Singleton mode starts the target release in a hardened transient systemd unit on a Singleton mode starts the new release in a hardened transient systemd unit on a
separate loopback address and probes it. The unit is named from the service and separate loopback address, probes it, stops the candidate, changes the current
a fresh bounded operation ID; a separate strict candidate-lease file binds that release pointer, and restarts the installed singleton unit.
unit to the candidate release, address, and start time without changing the
schema-1 deployment-state contract used by retained recovery binaries. Tend
then validates and atomically
routes the imported Caddy handler to that candidate. While the candidate serves
the canonical origin, Tend changes the current release pointer, restarts and
probes the installed fixed-address unit, validates Caddy again, and routes back
to it. The candidate remains healthy through the activation window and stops
only after the handoff succeeds.
After the local post-activation probes, Tend also checks configured HTTPS public Any failure before state persistence restores the previously observed Caddy
origins throughout a bounded activation window. Blue/green mode also probes the bytes and/or release pointers. Rollback is a separate explicit command over the
previous slot, and singleton mode probes the handoff candidate, for health and recorded state. It rechecks health and readiness, but deliberately does not
readiness throughout that window. Any failure before success is recorded apply the current release's content markers to an older release whose routes
restores the previously observed Caddy bytes and/or release pointers. State may differ. Pruning preserves both active and previous releases.
records desired, candidate, active, previous, and last-attempt
release identities, including failed attempts without claiming they became
active. Rollback is a separate explicit command
over the recorded state. It rechecks local health/readiness and public reachability,
but deliberately does not apply a future release's content marker to an older
release whose routes may differ. Pruning preserves both active and previous releases.
`tend reconcile` is a read-only observation boundary. It compares persisted
state, systemd activity, current/previous release pointers, and exact handler
file bytes. It does not claim to inspect Caddy's currently loaded in-memory
configuration, and it never clears or stops a candidate. The report makes
retained, inactive, ambiguous, and settled states legible before a future
explicit recovery operation is approved.
## Evidence boundary
Every attempted activation and explicit rollback emits bounded, versioned JSONL
events with an operation ID, service, approved artifact digest, source commit,
release version, phase, slot, elapsed duration, and outcome. Values are
validated rather than copied from command output. The log contains no
environment values, arbitrary process output, HTTP bodies, or secret paths.
Release-identity, event-file, and downstream observability failures are
deliberately best effort and cannot control Tend's deployment or rollback
result. A fresh operation identity is operationally required for a singleton
candidate unit; entropy failure therefore stops either forward activation or
rollback before a candidate process or traffic change.
+2 -219
View File
@@ -1,225 +1,8 @@
# Preview dogfood evidence # Preview dogfood evidence
This is maintainer-run operational evidence, not an independent audit or a This is maintainer-run operational evidence, not an independent audit or a
general reliability claim. It preserves each dated campaign and its limitations general reliability claim. It records the acceptance campaign completed on
instead of rewriting earlier observations as though later fixes had already August 14, 2026 before Tend's first public preview.
existed.
## Operation-scoped lease compatibility campaign — August 18, 2026
The candidate-lease compatibility fix completed trusted verification, exact
candidate reproduction, live activation, rollback, reactivation, and a bounded
failure injection on the production Linux/systemd/Caddy host. Conventional
deployment state remained schema 1 throughout. Singleton operation identity
used the separate adjacent lease introduced by this candidate.
### Assessed Tend candidate
- Private implementation source commit:
`789976ce766fd457ca10762540135e4e0e74cfe3`.
- Version: `v0.2.0-preview.2`.
- Linux/amd64 binary SHA-256:
`d0109bf037e493c7a046defe37818c3c1811806360b99c9a0910213665bd95c5`.
- Release-candidate archive SHA-256:
`2c225e0b9dd0be2d36fda62ab8d0b52cd9b28f9336c60dfb5edf3e715f450f95`.
- Toolchain: Go 1.26.6, `CGO_ENABLED=0`, `-trimpath`.
- Trusted verification run 366 and release-candidate run 369 passed for the
exact commit. The release workflow built the archive twice with identical
bytes and published checksums, build metadata, and an SPDX SBOM.
- The forge upload action reported a finalized 3.18 MB artifact, but both
documented artifact-list endpoints returned an empty result. A clean
independent clone therefore built the package twice and reproduced the
workflow's exact archive digest before deployment. This is recorded as an
artifact-publication limitation, not described as a successful consumer
download.
- The previous installed Tend binary remained retained by checksum before the
candidate replaced the active tool. Server policy validation then passed.
### Live maintenance and failure evidence
Gamertan activated archive
`c17b4db1a4e2fd5406b392ec72195b947272097e4d1bfcec9d700d0889c3a4c6`,
rolled back to its recorded previous archive, and reactivated the first archive.
Both blue/green units remained active with zero restarts. Reconciliation was
settled after every transition.
The Sandwich Hime website performed the same sequence with archive
`d418f93ced3307fa788f5d2209b5f09f16bceabec4a9698a9beec6a05e439f35`.
Every successful forward and rollback operation removed its operation-scoped
candidate lease and transient unit. A final intentional local-smoke failure
asked the otherwise valid candidate for an impossible marker. The candidate
failed before Caddy or release-pointer mutation, stopped, removed its lease,
recorded the failed attempt without changing the active release, and reconciled
as settled and consistent.
After the campaign, Gamertan, its news and case-study indexes, Sandwich Hime,
its documentation and tutorial, the Gamertan-mounted Sandwich route, and EQL
health all returned HTTP 200. Caddy, both Gamertan slots, and the installed
Sandwich Hime service were active with zero restarts and no warning-or-higher
journal entries during the campaign.
Observatory is not claimed by this candidate campaign. Its current dogfood
deployment is a Docker Compose singleton, which remains outside Tend's
versioned systemd/Caddy strategies. The earlier three-service campaign remains
valid for its assessed candidate, but the current commit must not be tagged
until Observatory either returns to a supported topology or a separately
reviewed Compose strategy completes the same activation, rollback, and failure
gates.
## Final Preview 2 code-candidate campaign — August 18, 2026
The final v0.2 Preview 2 implementation candidate completed two explicit
rollback-and-reactivation cycles for Gamertan, the Sandwich Hime website, and
Gamertan Observatory on the production Linux/systemd/Caddy host. The candidate
routed singleton traffic to the already-proven transient process while the
fixed-address installed unit restarted, then restored the canonical upstream
only after loopback and public-origin validation.
### Assessed Tend candidate
- Private implementation source commit:
`1fd3b9904c46e817c244196dd5d5a90921ca81a2`.
- Version: `v0.2.0-preview.2`.
- Linux/amd64 binary SHA-256:
`adb4753d4e865775d50d618c999f10dfac9a0de945ccfd9d0b8f2e80d65f4597`.
- Gitea release-candidate archive SHA-256:
`2bdfee2168cb16883d524cf9703adfa6ed5bc2bf44fbbb896993acc5fe6969ec`.
- Toolchain: Go 1.26.6, `CGO_ENABLED=0`, `-trimpath`.
- Trusted verification run 286 and release-candidate run 287 passed. The
archive's checksums, SPDX SBOM, embedded version, commit, clean VCS state,
target, and Go build information were independently rechecked before host
installation.
### Application maintenance artifacts
- Gamertan archive SHA-256:
`4700b075640b8b2fb5c17e0e02cf8d96ee67ceee10fe76d108c8b411983a88aa`.
- Sandwich Hime website archive SHA-256:
`46b4da41cf6703fb8818e7d25e3a9c13e57cf700f5608b1888c4adb3352e4d38`.
- Observatory preview 12 archive SHA-256:
`9ef0ddd8ec25d8fb75d6a6887e3ba874df7ebba16d3254f6250fcc646f4fd7f4`.
Every service ended with the intended current release active and the older
release retained as the explicit rollback target. Gamertan returned to its
green slot; both singleton services returned to their fixed addresses.
Candidate ports and the shared lock were free afterward. Caddy and all five
installed application units were active with zero restarts and no failed
units.
### Continuity and deployment evidence
A seven-minute workstation probe sampled the Gamertan origin, both Sandwich
Hime origins, Observatory, and EQL health 1,606 times each throughout the
campaign. It observed no HTTP failure status. One simultaneous client-side
disconnect affected all five destinations during a reload. A controlled
90-second replay therefore observed the same validated no-content-change Caddy
reload from both the workstation and an independent Linux host. The Linux host
recorded 450 successful responses and zero failures for every origin; the
workstation alone repeated one common-mode URL transport error across every
destination. Caddy retained the same PID with zero restarts and continued
serving unrelated requests. The common-mode workstation event is recorded as
an observer-path limitation, not server downtime.
The authoritative deployment-event files finished at 18 Gamertan events, 14
Sandwich Hime events, and 16 Observatory events. Observatory's agent cursor for
each stream exactly equalled the corresponding file size, proving complete
consumption. The agent and applications reported no warning-or-higher journal
entries during the campaign. Representative public routes returned HTTP 200,
and the EQL origin remained healthy.
### Findings closed before this campaign
Two earlier pre-activation artifacts exposed a mode-restoration defect under a
hardened root umask. They failed before route or pointer mutation. Tend now
reapplies validated archive modes explicitly and tests extraction under umask
`0077`.
The older singleton strategy briefly exposed an unavailable fixed upstream and
produced transient Observatory-agent `502`s. The final candidate's routed
handoff removed that failure in repeated production activation and rollback.
This evidence update changes VCS build metadata but not deployment logic. The
release policy therefore still requires one last maintenance pass with the
exact evidence-bearing candidate before the signed public tag is created.
## Restricted multi-service campaign — August 16, 2026
Tend's v0.2 implementation candidate completed the same maintenance contract
through the restricted transport for two independently configured services on
one Linux/systemd/Caddy host.
### Assessed Tend implementation
- Implementation source commit:
`840b77da708bbcd87a3203fb6a1f99b2984b8667`.
- Linux/amd64 candidate binary SHA-256:
`b3961315288871fa6085bf3b75c784a825a88f2bc3dd0694ad5d6c590eddf895`.
- Candidate archive SHA-256:
`948bad5271dca08b9445c387c5aea7f58f22add6b47b9071e1801a756aa71259`.
- Toolchain: Go 1.26.6, `CGO_ENABLED=0`, `-trimpath`.
- Trusted Gitea verification run 119 and release-candidate run 120 passed for
the exact implementation commit. CI and an independent, network-disabled
build each produced the same archive digest twice.
- The installed receiver accepted only a pinned Ed25519 host key, a dedicated
forced-command deployment key, the exact `tend-receive-v1` protocol, two
allowlisted service names, and separately repeated artifact digests. An
attempted arbitrary SSH command was refused.
- Production source, Go caches, repository credentials, secret values, remote
paths, and shell commands did not cross the transport boundary.
Adding this evidence changes VCS build metadata but not executable logic. The
signed preview tag and attached release evidence therefore identify the final
evidence-bearing candidate and its required last maintenance pass.
### Gamertan blue-green campaign
- Application source commit:
`a7e54047d3dc11671824b6ecc8ed698a9dd04421`.
- Preview 27 artifact SHA-256:
`edadef3a97c089771e2b6bb7dadd58928762284878f76f23b112c61ae282f17f`.
- The artifact was built twice, byte-identically, in the pinned Go 1.26.6
image with networking disabled and the audited cached module graph.
- The inactive slot passed health, readiness, page-marker, Caddy validation,
public-origin, Sandwich Hime mount, and EQL continuity checks before traffic
moved.
- Explicit rollback restored preview 26; its readiness and public boundary
passed; preview 27 was then reactivated through the recorded state.
- Sandwich Hime's release pointer and service remained unchanged throughout.
### Sandwich Hime singleton campaign
- Application source commit:
`435880c6751b773b6c5ee3ae6833d26e8eb7c0df`.
- Preview 30 artifact SHA-256:
`bee8d5bcde2c3c2e8bb96d5909a19889fc7f0be9390046c243fe20c9ff2ca44b`.
- The artifact was built twice, byte-identically, in the same pinned,
network-disabled Go 1.26.6 environment.
- A transient DynamicUser candidate passed health, readiness, tutorial marker,
canonical-origin, and Gamertan-mounted checks before the singleton pointer
and installed service changed. The candidate port was released afterward.
- Explicit rollback restored preview 28; its readiness and public boundary
passed; preview 30 was then reactivated through the recorded state.
- Both Gamertan slots, its selected Caddy upstream, and EQL remained healthy.
### Findings resolved by dogfood
The first restricted transfer stopped before artifact validation because
`sudo` removed `SSH_ORIGINAL_COMMAND`. The forced account still refused the
request; no service state changed. The 0BSD sudoers template now preserves only
that one server-supplied variable, while the receiver requires its exact
protocol value and rejects every other command.
The first singleton candidate stopped before pointer mutation because the
shared environment file's live listen value overrode the candidate address.
The site remained on its former release and retained the same process. Tend now
rejects singleton shared environment files containing the configured listen
key; installed units own the non-secret live address and Tend supplies only the
transient candidate address.
Final verification found both service states valid, every installed unit and
Caddy active with zero restart failures, the candidate port closed, no warning
or error entries after the successful campaign, and representative Gamertan,
Sandwich Hime, mounted, and EQL routes returning HTTP 200.
## Assessed candidate ## Assessed candidate
-656
View File
@@ -1,656 +0,0 @@
# Dogfood friction ledger
This document records friction observed through August 19, 2026 while using
Tend for real maintenance releases. Friction is evidence about the product: it
should become either a clearer contract, safer automation, or an explicit
non-goal. It must not become application-specific shell lore.
The findings below are maintainer observations, not implemented promises. Tend
continues to fail closed while an option is being designed. In particular,
there is no `--skip-remote` packaging escape hatch, no arbitrary deployment
hook, and no relaxation of the host-wide activation lock.
## Design standard
A Tend workflow should make the secure path the short, documented path:
- source, artifact, approval, configuration, and active-release identities are
explicit and inspectable;
- credentials are narrowly scoped, briefly available, and absent from
artifacts and logs;
- application validation is declarative, bounded, and cannot become a shell;
- first installation, maintenance, activation, rollback, and pruning are
distinct operations;
- every mutation has a recorded prior state and a tested restoration path;
- Tend-owned state is an evidence journal and reconciliation aid, not an
exclusive claim over an otherwise conventional systemd/Caddy service;
- tool-owned transient resources are leased, inspectable, resumable, and safe
to reconcile without requiring operators to understand internal names;
- unrelated services may build and prepare candidates concurrently, while the
shared Caddy activation boundary remains serialized.
## Observed findings
| ID | Status | Finding | Current safe behavior | Candidate direction |
| --- | --- | --- | --- | --- |
| F-01 | Workflow mitigation | CI packaging needs proof that the exact commit was pushed. | Packaging verifies the configured remote and fails if Git cannot authenticate. The trusted workflow retains only its job-scoped read-only checkout credential. | Make the proof credential or an authenticated source attestation an explicit Tend input and evidence boundary. |
| F-02 | Open | Application-owned unit and configuration changes are outside the binary activation transaction. | Operators stage, validate, back up, and restore those files separately. | Add an allowlisted configuration transaction or record and validate exact configuration digests. |
| F-03 | Open | A singleton candidate may not reproduce the installed unit's arguments and application configuration. | It receives the production environment file and an isolated listen override; activation still fails closed. | Add a bounded application preflight and explicit, validated candidate invocation. |
| F-04 | Partially resolved in development | Artifact production, review, approval, transfer, and activation require several identity checks. | v0.2 standardizes the evidence bundle and restricted transfer; `tend inspect` now exposes the complete non-mutating artifact validation as a named workflow. | Preserve explicit approval while adding digest-bound publication receipts. |
| F-05 | Partially resolved | Tend maintenance assumes an existing adopted service and current release. | `check-server` validates a prepared host, while first installation remains a separately reviewed operator procedure. | Define an explicit `install` or `adopt` transaction rather than silently treating bootstrap as maintenance. |
| F-06 | Partially resolved | Binary rollback can be unsafe when service configuration has changed incompatibly. | v0.2 records desired, candidate, active, previous, and last-attempt releases, but external configuration compatibility is operator-owned. | Bind non-secret configuration identities and preflight results to release state. |
| F-07 | Open | `GOPROXY=off` does not prove that every module metadata lookup is available locally. | Packaging stops before artifact creation when Go cannot resolve the complete pinned module graph. | Separate checksum-verified dependency resolution from a network-disabled, cache-completeness-checked build stage. |
| F-08 | Partially resolved in development | Packaging an otherwise clean pushed commit from a linked Git worktree fails Go's required VCS-status stamp. | Tend now detects a linked worktree before remote proof and build work and gives an exact standalone-clone instruction. | Assess provenance-preserving linked-worktree support without weakening `-buildvcs=true`. |
| F-09 | Resolved in Preview 2 | Restarting a fixed-address singleton briefly exposed Caddy to an unavailable upstream. | Tend now routes the imported handler to the candidate, restarts and proves the installed unit behind that handoff, then routes back while the candidate remains healthy through the activation window. | Preserve the production regression campaign and failure-injection matrix. |
| F-10 | Resolved in Preview 2 | A hardened root umask could make an extracted application binary executable only by root. | Tend reapplies every validated archive mode explicitly; extraction is tested under umask `0077`. | Preserve the mode and non-root candidate tests. |
| F-11 | Evidence practice | A single external observer can report common-mode client or network errors as apparent multi-service downtime. | Production campaigns retain server state and use an independent observer before classifying a continuity failure. | Standardize multi-vantage continuity evidence without making an observability dependency part of deployment authority. |
| F-12 | Partially resolved in development | A deliberately retained singleton handoff candidate could occupy Tend's fixed transient-unit name and block the next valid activation. | Candidates now have operation-scoped unit identities and bounded persisted leases; `reconcile --json` reports release identity, unit, pointer, and handler-file facts without mutation. | Add an explicit, idempotent `resume`/repair operation only after its route-identity and failure-injection model is proven. |
| F-13 | Open | A configured public-origin probe may resolve to a different deployment after a DNS or edge migration. | Operators separately verify local routed-origin identity and external DNS topology before activation. | Bind probes to expected release and edge identities; classify topology drift instead of treating a body marker as deployment proof. |
| F-14 | Open | An application can bind durable data identity to an absolute path that a binary-only candidate never exercises. | Preserve the production path and run application-owned validation before traffic moves. | Add a bounded, explicitly read-only application preflight and record which configuration/data identities it assessed. |
| F-15 | Product direction | Requiring Tend-specific residue and metadata to be perfect can make the facilitator feel like an exclusive deployment owner. | Conventional systemd/Caddy recovery remains authoritative and every manual intervention is captured as evidence. | Make state append-only and migratable, infer observed state safely, and keep Tend removable without making the service obscure or undeployable. |
| F-16 | Open | A reusable Docker network alias can identify both the live service and a retained handoff candidate, making the routed target ambiguous during activation. | Pin the temporary Caddy handoff to the exact, observed candidate address and restore the reviewed application handler after activation. | Allocate operation-scoped network identities and reject any candidate or handler target that resolves to more than one container or release identity. |
| F-17 | Open | A CI job can report a successful artifact upload while the forge artifact API exposes no retrievable artifact to the approval/deployment client. | Reproduce and verify the exact pushed commit locally, then compare two packages byte-for-byte before approving the digest; never guess an artifact URL. | Define a digest-bound artifact handoff with an independently readable receipt and fail the workflow unless the approval client can retrieve and inspect the exact uploaded bytes. |
| F-18 | Open | Tend's current strategies do not model a Docker Compose singleton, so a manual Observatory replacement briefly exposed an unavailable upstream. | The exact image, data backup, rollback image, and health probes were preserved, but Compose replaced the only live container and an external observer saw a bounded `502`/`503` window. | Design an explicit Compose strategy with operation-scoped candidates, unambiguous routing, durable-data preflight, activation continuity, and automatic rollback; do not add arbitrary container hooks to the systemd strategies. |
| F-19 | Open | A successful application schema migration can make the previous binary unreadable even when its image and service definition are intact. | Preserve verified database backups and prefer completing the proven forward activation; do not execute a binary-only rollback after a forward-only migration. | Make applications declare schema compatibility and an explicit data restoration or forward-recovery plan before Tend offers automatic rollback. |
| F-20 | Open | A stateful migration can require bounded temporary scratch space that a stateless binary candidate never exercises. | Run the exact candidate against a copied production data set under its proposed resource limits; record the reviewed scratch budget in the service definition. | Bind resource/configuration digests to the candidate and add an application-owned stateful preflight contract without arbitrary hooks. |
## F-01: pushed-commit proof in CI
### Observation
During an Observatory release-candidate run, verification intentionally checked
out source without persisted credentials. Tend later attempted to prove that
`HEAD` existed on the configured private Gitea branch and Git could not
authenticate. Packaging stopped before producing an artifact. This was the
correct fail-closed outcome, but the credential lifecycle was not obvious from
the workflow contract.
The trusted workflow uses a job-scoped read-only checkout credential so Tend
can perform the independent remote proof. It is not a long-lived repository or
deployment credential, and it is removed by checkout cleanup at the end of the
job. Production credentials remain unavailable to the build job.
### Options to assess
1. Support an explicit read-only Git credential file or credential helper for
package-time proof. Never accept a token in arguments, configuration,
manifests, artifacts, or logs.
2. Accept an authenticated CI source-attestation document binding repository,
branch, commit, tree, workflow identity, and event identity. Define which
CI issuers are trusted and preserve the attestation with release evidence.
3. Accept a signed, pre-verified source bundle whose identity and policy can be
checked without network access.
Do not add a generic skip flag. An unavailable proof must remain a packaging
failure unless an equally strong proof mode was selected explicitly.
## F-02: application configuration is not binary activation
### Observation
Tend can transact an immutable binary, release pointers, a systemd restart, a
validated Caddy handler, health checks, and rollback state. It does not
currently transact application configuration, systemd unit changes, credential
bindings, or server-local secret files. A first deployment that changes these
files therefore needs a separate backup, validation, installation, and
restoration procedure.
This boundary is safe but easy to overlook: restoring the previous binary does
not restore an incompatible unit or application configuration.
### Options to assess
1. Add strict managed-file entries with exact source and destination paths,
content digest, owner, group, mode, and validation type. Permit only regular
files beneath configured roots; reject symlinks and unknown destinations.
2. Add a separate `tend configure` transaction that backs up, atomically
replaces, validates, and restores supported systemd/Caddy/application files.
3. Keep configuration externally managed, but require Tend to record desired
and active configuration digests and prove candidate/rollback compatibility.
None of these options should permit arbitrary shell commands. Secret values
remain referenced server-side and must never enter release artifacts or state.
## F-03: candidate fidelity
### Observation
A hardened singleton candidate receives the configured production environment
file and Tend's isolated listen-address override. It does not currently model
the installed systemd unit's complete argument vector. An application whose
configuration path is supplied by unit arguments may therefore start a
candidate with defaults instead of the intended production configuration.
That preserves live-state isolation, but it may prove only executable startup.
A missing credential, incompatible configuration field, filesystem permission,
or data migration requirement can then surface at activation time.
### Options to assess
- Define a fixed application preflight command or protocol that validates the
production configuration, credentials, permissions, and data compatibility
without binding the live port or mutating live state.
- Add a strict candidate argument vector to schema 2. Validate each argument,
reject secret values and paths outside the application contract, and pass it
directly to systemd without a shell.
- Permit an allowlisted `check` argument vector, never a shell string, with
bounded time/output and an explicitly non-mutating application contract.
- Record which checks ran against the binary alone and which ran against the
actual production configuration so the evidence cannot overstate coverage.
## F-04: artifact review and approval ergonomics
### Observation
The secure flow deliberately separates build, review, digest approval,
transfer, candidate validation, and activation. v0.2 now produces a consistent
archive, manifest, SBOM, and checksum set, then transfers one exact artifact
through its restricted receiver. In practice, operators still need a
predictable way to discover that CI artifact, inspect it, approve exactly one
digest, and retain the review evidence. Without a first-class review path,
correct manual steps are easy to reconstruct differently for each application.
### Options to assess
- Add `tend inspect` for offline, non-mutating verification and a concise human
and JSON summary of source, build, dependency, and archive identities.
- Add an approval record that binds the artifact digest, service, target,
approver, and expiry without containing a credential.
- Standardize one Gitea artifact layout and documented download-to-activation
workflow. Keep production credentials unavailable to verification jobs.
Approval must remain explicit; better ergonomics must not turn a successful
build into an automatic production mutation.
## F-05: first installation and adoption
### Observation
The maintenance workflow expects an installed service, valid configuration,
and a current release pointer. `check-server` validates the restricted receiver
policy and prepared service boundary, but it does not create them. Observatory
bootstrap therefore required operator-managed service account, directories,
credentials, unit, configuration, and an initial current release before Tend
could own later maintenance safely.
### Options to assess
- `tend install`: a deliberately broader, separately approved transaction with
a strict schema and complete rollback of every supported created object.
- `tend adopt`: validate an existing service and release, copy or identify its
immutable artifact, establish state, and refuse ambiguous ownership.
- Keep bootstrap out of Tend, but ship a versioned acceptance checklist and a
machine-readable `check-server` result that maintenance can require.
Installation and adoption must not be inferred from a missing state file.
## F-06: release and configuration identity
### Observation
Content-addressed releases and v0.2 state make desired, candidate, active,
previous, and last-attempt binary identities clear. A live service is still a
combination of its binary, application configuration, credential bindings,
unit, routing fragment, and sometimes data schema. Tend cannot yet fully
explain that combined identity or determine whether a retained binary is
compatible with the current external configuration.
### Options to assess
- Extend the existing desired, candidate, active, previous, and last-attempt
release state with non-secret configuration and unit digests.
- Require rollback compatibility declarations or read-only application
preflight before changing traffic.
- Expose the identities and last validation results through
`tend status --json` for deployment evidence and future Observatory
ingestion.
Configuration records contain digests and approved metadata only—not secret
contents.
## F-07: offline module-cache completeness
### Observation
An exact Observatory package attempt used `GOPROXY=off` and a previously used
module cache. Source archives for the application dependencies were present,
but Go still needed several module metadata records while Tend enumerated the
complete build graph. Go refused the lookup and Tend stopped before building
or writing an artifact. Re-enabling the checksum-verified public proxy supplied
the missing metadata; the resulting package was byte-identical to the trusted
CI candidate.
This is safe failure, but `GOPROXY=off` alone is not evidence of a hermetic
build. A cache can be partially populated even when ordinary builds happen to
succeed.
### Options to assess
- Add a resolver stage that runs with the pinned toolchain, proxy, and checksum
database, emits the complete module inventory, and materializes a bounded,
read-only cache for the builder.
- Run Tend's package stage with networking disabled and require every module,
checksum, source archive, and metadata record to come from that reviewed
cache.
- Add a non-mutating cache-completeness check that reports missing module
identities before the expensive double build.
- Consider a strictly verified vendored-source mode where repository size and
update review are acceptable; do not silently change dependency modes.
Do not treat `GONOSUMDB`, `GOPRIVATE`, or a proxy bypass as an offline-build
solution. They alter verification or routing policy rather than proving cache
completeness.
## F-08: linked-worktree VCS stamping
### Observation
An exact clean, pushed Observatory main commit passed verification and public
snapshot isolation from a detached linked Git worktree. Tend accepted its
source and remote provenance, then Go 1.26.6 stopped both `tend package` and an
equivalent direct build at the required `-buildvcs=true` step with `error
obtaining VCS status: exit status 128`. Ordinary Git status and commit queries
from the same worktree succeeded.
A fresh standalone SSH clone of the identical commit packaged successfully
twice. Both archives were byte-identical and carried the expected commit, Go
version, VCS settings, checksums, manifest, and SBOM. This isolates the failure
to the linked-worktree build shape rather than the application source or module
graph.
The failure is safe: Tend did not create a partial artifact and must not switch
to `-buildvcs=false`, because the package gate independently verifies the
embedded VCS revision and clean state. Until the interaction is resolved, use
a clean standalone clone for release packaging.
### Options to assess
- Detect a `.git` indirection file during `tend check` and fail before the
expensive double build with a precise standalone-clone instruction.
- Reproduce the interaction in a package integration test against the minimum
supported Go release and determine whether it is a Go toolchain limitation
or an invocation/environment defect.
- If linked worktrees can be supported, require the resulting build record to
carry the exact expected `vcs.revision` and `vcs.modified=false`; do not
synthesize those settings or disable VCS stamping.
- Consider an explicit, signed source-bundle input as part of the broader
source-attestation design. It must remain at least as strong as current
pushed-commit proof.
## F-09: singleton traffic continuity
### Observation
During the August 18 Observatory maintenance exercise, the transient candidate
passed health, readiness, and content checks. Tend then stopped that candidate,
changed the singleton pointer, and restarted the installed fixed-address unit.
Caddy still targeted the fixed address during that restart, so the Observatory
agent observed a small number of transient HTTP `502` responses. Its durable
spool retried successfully and no accepted telemetry was lost, but the routed
origin was not continuously available.
The corrected activation contract treats the candidate as a traffic handoff,
not merely a preflight process. Tend validates and routes the imported Caddy
handler to the candidate before changing the release pointer. It restarts and
probes the fixed-address unit without public traffic, routes back only after
that unit passes, and keeps the candidate healthy throughout the bounded
activation window. Any failure restores the former pointer and exact handler
bytes. If restoration itself cannot be completed, Tend leaves the proven
candidate routed and running for explicit operator recovery instead of causing
a known outage.
Regression tests cover successful handoff, restart failure, public-origin
failure during the activation window, pointer and handler restoration, Caddy
validation/reload boundaries, and candidate cleanup. Final production evidence
must still repeat deploy, rollback, and reactivation with the exact release
binary before this finding is treated as released.
## F-10: archive modes under a hardened umask
### Observation
The first two August 18 maintenance candidates stopped before activation. The
release archives correctly recorded executable mode `0755`, but extraction by
root under umask `0077` left the installed application binary mode `0700`.
The unprivileged transient candidate could not execute it. Tend emitted failed
candidate evidence, retained the prior release and route, and did not expose
the failed binary to traffic.
Extraction now reapplies the already validated archive mode after file content
is closed. A Linux regression test sets umask `0077`, extracts the release, and
requires the installed binary to remain executable by the service identity.
The successful maintenance campaign used that corrected Tend binary.
## F-12: retained-candidate reconciliation
### Observation
An Observatory activation failure intentionally left its proven handoff
candidate routed and running for operator recovery. A later push transferred
and validated a new content-addressed artifact, then stopped before candidate
startup because systemd still had Tend's fixed candidate unit loaded:
`Unit observatory-tend-candidate.service was already loaded or has a fragment file.`
The artifact, application, and host were valid. The collision was Tend-owned
residue from its own safe fallback behavior. Recovery required starting the
same stateless fallback under an independently named hardened unit, proving it,
routing Caddy to it, and only then stopping the old candidate and confirming
its port and unit name were free. No accepted application state was lost, but
the operator had to understand Tend's internal unit convention.
### Implemented foundation
- Transient candidate units are named by service plus the first 12 hexadecimal
characters of a fresh 128-bit operation identity.
- Conventional deployment state remains schema 1 so the retained v0.1 binary
can still read active and previous release identities. Singleton operations
write a separate strict, adjacent candidate-lease file containing the
operation, release, unit, address, and start time.
- `tend reconcile --json` reports persisted state, installed/candidate unit
activity, release pointers, and whether the exact handler file matches the
installed or candidate upstream. It explicitly distinguishes handler file
bytes from Caddy's loaded runtime configuration and performs no mutation.
- A retained forward-activation or rollback candidate lease survives an
incomplete recovery instead of being erased by a generic failed-attempt path.
Older binaries ignore the additive lease file, so operators must reconcile
before downgrading or attempting another activation.
### Remaining options to assess
- Extend the candidate lease with separately observed health and loaded-route
evidence rather than inferring either from a successful file write.
- Add an explicit `tend resume` operation that distinguishes active, routed,
rollback, abandoned, and unknown candidates without mutating by default.
- Permit automatic cleanup only after Tend proves that Caddy, current/previous
pointers, and the installed unit do not reference the candidate and that a
healthy route remains.
- Make a repeated activation of the same approved digest idempotently resume
the recorded operation instead of restaging or colliding with itself.
Never resolve the collision by blindly stopping the loaded unit. A retained
candidate may be the only healthy route after a failed singleton activation.
## F-13: public-origin topology and release identity
### Observation
During the same recovery, the public DNS record had already moved to a new
edge while the old host's Tend policy still named the canonical HTTPS origin
as its post-activation smoke target. A request from the old host would therefore
test a different deployment. It could fail even when the old host was healthy,
or pass against a matching marker served by the new host. Neither result proves
the release that Tend just activated.
### Options to assess
- Separate a local routed-origin probe—Caddy with the canonical Host/SNI on the
deployment host—from an external DNS-origin observation.
- Bind local success to a non-secret application release identity such as the
approved artifact digest, commit, and version, not only a human page marker.
- Record the expected public edge identity or resolved address set at approval
time. If it changes, classify the result as topology drift and require an
explicit migration decision instead of reporting an application failure.
- Allow independent external observers as additional evidence, but never let
a response from an unidentified deployment authorize activation.
Public reachability remains valuable; it must be evidence about the intended
deployment rather than merely evidence that the hostname answered.
## F-14: application data-path and preflight fidelity
### Observation
An exact, checksummed Observatory data copy was mounted at a different absolute
path in a container. Raw objects and durable SQLite databases were byte
identical, but the application catalogue intentionally compared stored segment
paths with filesystem-derived identities and failed closed. Preserving the
original in-container data path made the same Preview 15 binary ready in four
seconds with zero restarts and bounded memory.
This was an application portability constraint, not a reason for Tend to
rewrite database state. It also demonstrates the limit of a binary-only
stateless candidate: executable health cannot prove compatibility with the
real configuration, credentials, mounts, or durable data.
SQLite `-wal`, `-shm`, and process-lock files also changed during ordinary
open/close behavior while raw segments and durable databases did not. Migration
evidence should distinguish durable application truth from ephemeral runtime
coordination files.
### Options to assess
- Support a fixed, bounded application-owned preflight argument vector with an
explicit non-mutating contract, timeout, output limit, and no shell.
- Record the exact non-secret configuration digest, data-root identity, mount
identity, and checks performed alongside binary candidate evidence.
- Let applications define their durable migration evidence set; Tend should
transport and report those digests but must not infer database semantics or
edit state.
- Require rollback compatibility to be assessed against the same configuration
and data identities before an older binary is called safe.
## F-15: facilitator rather than owner
Tend's strict boundaries are valuable where authority changes hands: source
provenance, artifact digest approval, restricted transport, secret references,
path validation, traffic movement, and rollback. Strictness becomes harmful
when Tend's internal names or stale metadata are treated as application
requirements that an authorized maintainer must reverse-engineer.
The target model is less invasive:
- systemd units, Caddy handlers, environment files, release directories, and
application checks remain conventional and independently operable;
- Tend records an append-only migration/deployment journal and derives an
observed state before proposing mutation;
- schema migrations preserve old records and explain compatibility rather than
silently rejecting safe, recognizable state;
- `check`, `status`, and `reconcile` show facts and proposed repairs without
mutation; approval is required for artifact selection and traffic movement,
not for Tend implementation trivia;
- installation and adoption are explicit, while uninstalling Tend leaves an
understandable, runnable service and complete evidence trail.
This does not relax hostile-input defenses. It moves strictness to the trust
boundary and makes recovery humane for the authenticated operator.
## F-16: ambiguous container-network identity
### Observation
During the Observatory Preview 16 activation, the retained stateless handoff
container and the live Compose service both answered to the Docker network
alias `observatory`. A Caddy target such as `observatory:8093` could therefore
resolve to either release. Removing or restarting one container could also
change which release received traffic. The application and both containers
were healthy; the ambiguity existed solely in deployment identity.
The activation used a separately validated Caddy fragment pinned to the exact
candidate address, proved its release marker before traffic moved, replaced
the live service, and then restored the reviewed application handler. No
ambiguous alias was used for the handoff.
### Options to assess
- Give every candidate an operation-scoped container name, network alias, and
lease that bind directly to its artifact digest and expected address.
- Resolve and inspect a proposed upstream immediately before Caddy validation;
reject zero, multiple, or identity-mismatched targets.
- Prefer a dedicated candidate network or an address supplied by the container
runtime over a stable alias shared with the live service.
- Record the exact routed target and release proof in activation state, then
verify that the restored production handler identifies the active release.
Do not treat a healthy response from an ambiguous service name as release
proof. Availability and deployment identity are separate properties.
## F-17: CI artifact publication is not artifact availability
### Observation
The trusted Observatory release-candidate workflow completed its two builds,
byte comparison, checksums, manifest, and SBOM, and its upload action reported
success. The forge artifact API subsequently returned no artifact for that
run. The deployment client therefore had no independently discoverable object
to download and inspect. Guessing a web-interface route or treating a green
upload step as possession of the bytes would have weakened the build-approve-
deploy boundary.
The release was instead reproduced twice from a fresh, clean clone of the
exact pushed commit with the pinned toolchain. The packages compared
byte-for-byte, their manifest, checksums, SBOM, version, commit, and tree were
verified, and the approved digest was recorded. This preserved release
identity, but it is an operator workaround rather than the desired CI handoff.
### Options to assess
- Require the publisher to return a versioned receipt containing forge, run,
artifact identifier, size, digest, retention, and retrieval endpoint.
- Add a separate read-only verification step that downloads the artifact using
the same interface available to the approval client and reruns `tend
inspect` before the workflow is considered publish-complete.
- Support a content-addressed, append-only artifact store whose object name is
the approved SHA-256 and whose credentials remain separate from production.
- Let `tend push` accept only a locally present artifact plus an optional
verified publication receipt; never allow a successful CI status alone to
select bytes for deployment.
Artifact availability must be proven from the consumer side. A successful
upload log is evidence of an attempted publication, not evidence that the
approved bytes can be recovered.
The same failure repeated for Tend release-candidate run 369: the pinned job
successfully built identical archives, recorded digest
`2c225e0b9dd0be2d36fda62ab8d0b52cd9b28f9336c60dfb5edf3e715f450f95`,
and finalized a 3.18 MB upload, while both repository-wide and run-scoped REST
artifact listings returned no objects. An independent clean build reproduced
the exact digest before dogfooding. This confirms the problem is a reusable
forge-to-approval gap rather than an Observatory-specific packaging defect.
## F-18: Docker Compose is not a systemd singleton
### Observation
Observatory Preview 18 was packaged reproducibly and its exact scratch image
passed a constrained loopback candidate check. The production service on
cliff-mads is a Docker Compose singleton, while Tend's supported strategies are
systemd/Caddy blue-green and systemd singleton-candidate. Tend was therefore
not used to pretend that an unsupported runtime had received a complete Tend
activation proof.
The manual Compose replacement retained the former image and online database
backups and completed health, readiness, route, projection, agent, and log
checks. An independent public observer nevertheless recorded `200`, then
`502`/`503`, then `200` across an approximately 31.6-second replacement and
startup window. No application restart loop or accepted-data loss was
observed, but this is not continuous delivery.
Preview.19 strengthened this finding. The exact candidate passed direct health
and readiness, but an activation script treated the first transient routed
`503` as final and attempted rollback before the proxy path had settled. The
previous binary then correctly rejected the newly migrated control schema and
entered a fail-closed restart loop. The operator recovered with the already
proven forward candidate while the agent remained paused. This was a bounded,
observable failure with intact backups, but it demonstrates that proxy
settling and data-schema rollback are different gates and cannot share one
generic failure branch.
### Options to assess
- Add a distinct, versioned Docker Compose strategy rather than arbitrary
command hooks or hidden container behavior inside existing strategies.
- Allocate an operation-scoped candidate container and network identity that
cannot collide with the production alias.
- Validate the exact image digest, non-secret Compose/configuration digest,
mounts, durable data identity, and application-owned read-only preflight
before moving traffic.
- Route Caddy to the proven candidate, replace or promote the production
service while the candidate remains healthy, then restore the reviewed live
handler only after the final container proves its release identity.
- Restore the prior handler and retained image automatically when any
activation-window probe fails; keep data rollback an explicit
application-owned decision.
This strategy must remain optional. Tend should facilitate a conventional
Compose service without requiring Tend-only container labels, aliases, or
state for ordinary operator recovery.
## F-19: binary rollback is not data rollback
### Observation
Observatory Preview.19 migrated its live control database from the schema
understood by Preview.18 to schema 11 before activation. When a premature
public-origin failure triggered binary rollback, Preview.18 rejected the new
schema rather than interpreting unknown state. That fail-closed behavior was
correct, but Tend-like orchestration cannot infer from two binary identities
whether their durable schemas are mutually readable.
The exact pre-migration control and projection databases had already been
copied, integrity-checked, mode-restricted, and SHA-256 verified. The new
candidate had also passed the migration on an isolated copy. Because the
migration was forward-valid and the old agent was paused, completing the
Preview.19 activation preserved more verified state than restoring the backup.
### Options to assess
- Let an application declare a versioned, bounded compatibility statement for
current-to-candidate and candidate-to-previous data access.
- Treat migration completion as a journalled phase after which automatic
binary rollback is permitted only when backward readability was explicitly
proven.
- Support an application-owned, separately approved data-restoration plan;
never infer one from release pointers or run an arbitrary rollback hook.
- Keep verified backups and the proven forward candidate until the migration
soak closes, even when the previous binary remains retained.
## F-20: stateful preflight needs its real resource envelope
### Observation
The Preview.19 stateless candidate and ordinary tests passed, but building a
presence-only SQLite index over a copied production projection failed with
`database or disk is full` under the service's 64 MiB `/tmp` tmpfs. The exact
same candidate and data passed with a 512 MiB tmpfs and drained all pending raw
segments. The projection database was healthy; the one-time index sort needed
more bounded scratch space than steady-state operation.
### Options to assess
- Record the non-secret service configuration digest and resource envelope
beside the artifact digest.
- Permit a strict application-owned preflight command selected from a reviewed
schema, never an arbitrary shell string.
- Exercise migrations against a copied or snapshot-backed production data set
under the candidate's exact CPU, memory, temporary-storage, filesystem, and
credential boundaries.
- Report resource exhaustion distinctly from corrupt data, failed health, or
incompatible schema so recovery guidance remains accurate.
## Prioritization
The recommended implementation order is:
1. operation-scoped candidate leases plus explicit resume/repair built on the
new read-only reconciliation report;
2. migration compatibility and stateful-resource preflight contracts;
3. a bounded Docker Compose strategy with operation-scoped container/network
identities and unambiguous upstream
resolution and release proof;
4. release-bound local routed-origin proof separated from external DNS proof;
5. a bounded application preflight and configuration/data identity record;
6. a consumer-verifiable, digest-bound CI artifact publication receipt;
7. final production proof of the singleton traffic handoff;
8. a checksum-verified resolver and network-disabled package contract;
9. a provenance-preserving linked-worktree support decision after the new
early diagnostic;
10. a standardized CI artifact/approval contract building on `tend inspect`;
11. exact release plus configuration identity in status and state;
12. restricted transfer and receive with host policy;
13. explicit adoption for existing services;
14. managed configuration only after its restoration and failure-injection
model is as strong as binary activation.
Friction entries should be updated with the implementing version, tests, and
dogfood evidence when resolved. Resolved entries remain in this ledger so the
reason for the security boundary is not lost.
-5
View File
@@ -12,8 +12,3 @@ material markers. The resulting directory receives a new public root commit.
Public Gitea is canonical for issues, contributions, and releases. GitHub is a Public Gitea is canonical for issues, contributions, and releases. GitHub is a
read-only discovery copy of the same public tree. Tags belong only to canonical read-only discovery copy of the same public tree. Tags belong only to canonical
Gitea. Gitea.
The snapshot includes the program, security and architecture documentation,
copyable examples, release configuration, and local verification scripts.
Private workflows, runner configuration, repository credentials, and raw
operational evidence remain outside the public root.
-44
View File
@@ -1,44 +0,0 @@
# Schema 1 to schema 2
Configuration schema 2 and deployment state are separate versioned contracts.
The service configuration moves to schema 2, while conventional deployment
state deliberately remains schema 1 so the retained v0.1 binary can still read
active and previous release identities during recovery.
Singleton operations write an adjacent, strict
`<state-file>.candidate-lease.json` file containing the operation, release,
unit, address, and start time. Older binaries ignore that additive file. Do not
downgrade or start another deployment while a lease is present: first use the
new binary's `tend reconcile --json` report to establish which process and
route are healthy. Tend does not silently invent a lease for a legacy
interrupted operation.
1. Move each configuration to `/etc/tend/services/<service>.json`.
2. Set `schema_version` to `2`.
3. Add `service.environment_file` below `/etc/tend/environment/`.
4. Create that file as a regular root-owned file with mode `0600`. Move secret
values out of JSON. Do not put the environment file in Git. For a singleton,
keep its `listen_env` key out of this shared file: set the live address in
the installed unit and let Tend supply only the candidate address.
5. Set every service's `deployment.lock_file` to
`/run/lock/tend-deploy.lock`.
6. Add one or more query-free HTTPS `deployment.public_smoke` checks.
7. Add a per-service `deployment.event_log` below its release root and a
bounded `deployment.activation_window_seconds` value. Keep the log
root-owned and grant collectors read access explicitly.
8. For singleton services, add the full Caddy configuration, imported handler,
and one-upstream handler-template paths. The template must be reviewed and
contain exactly one `{{UPSTREAM}}` marker so Tend can keep traffic on the
candidate while the fixed-address unit restarts.
9. Update installed systemd units to read the same environment file as the
transient candidate.
10. Install a root-owned `0600` receive policy mapping each service name to its
exact configuration and artifact-size ceiling.
11. Run `tend check-server` as root before accepting a transfer.
12. Validate, activate, rollback, and reactivate one service at a time. Confirm
unrelated services never restart.
Tend does not discover `.env`, infer old values, rewrite a production file, or
silently migrate state. For local development, copy a committed `.env.example`
to an ignored `.env.local`, restrict its mode, and load it with the application's
own tooling. Tend never loads local dotenv files implicitly.
+14 -30
View File
@@ -7,49 +7,33 @@
- Release roots, state, pointers, and Caddy files reject symlink substitution - Release roots, state, pointers, and Caddy files reject symlink substitution
at their checked boundaries. at their checked boundaries.
- Configuration is strict JSON and is never interpolated into a shell command. - Configuration is strict JSON and is never interpolated into a shell command.
- Candidate health is established before traffic or any current release - Candidate health is established before traffic or the singleton current
pointer changes. New deployments also satisfy configured content smoke pointer changes. New deployments also satisfy configured content smoke
checks; rollback uses health and readiness because future-release content checks; rollback uses health and readiness because future-release content
markers are not valid requirements for an older retained release. markers are not valid requirements for an older retained release.
- Caddy configuration validates before reload. - Caddy configuration validates before reload.
- Canonical routed origins and the previous blue/green slot or singleton
handoff candidate remain under probe for the configured activation window; a
failure restores the old handler and release pointer.
- Desired, candidate, active, previous, and failed-attempt identities remain
distinct in state. Bounded deployment events contain no arbitrary command
output or environment values and cannot block deployment.
- An activation failure restores the previously observed state. - An activation failure restores the previously observed state.
- Active and previous releases survive pruning. - Active and previous releases survive pruning.
- The restricted receiver accepts one versioned bounded stream, one allowlisted
service name, and one explicitly approved digest. It accepts no remote path,
URL, environment value, or shell expression.
- OpenSSH host keys are pinned; forwarding, PTYs, proxy commands, local commands,
and user SSH configuration are disabled by the client invocation.
- Production secret values are absent from configuration, process arguments,
artifacts, manifests, reports, deployment state, and Tend logs.
- One host-wide lock serializes shared Caddy activation without stopping or
reconfiguring unrelated services.
## Trusted inputs ## Trusted inputs
Reviewed source, the pinned Go toolchain, root-owned server policy and service Reviewed source, the Go toolchain, root-owned `tend.json`, systemd unit files,
configuration, systemd unit files, the Caddy handler template, the operator, the Caddy handler template, the operator, and the target host's root account are
and the target host's root account are trusted. Packaging a hostile repository trusted. Packaging a hostile repository can execute hostile Go compiler hooks
can execute hostile Go compiler hooks or consume resources; Tend is not a or consume resources; Tend is not a source-code sandbox.
source-code sandbox.
## Adversarial inputs ## Adversarial inputs
Artifact paths and bytes, protocol frames, requested service names, archive Artifact paths and bytes, archive metadata, stale or malformed state, HTTP
metadata, stale or malformed state, HTTP responses, subprocess failures, and responses, subprocess failures, and filesystem objects at managed paths are
filesystem objects at managed paths are validated and fail closed. Process validated and fail closed. Process output is bounded.
output and receive fields are bounded.
## Non-goals and preview limits ## Non-goals and preview limits
Tend does not defend against a concurrently malicious root user, a compromised Tend does not defend against a concurrently malicious root user, a compromised
kernel/toolchain/systemd/Caddy/OpenSSH installation, denial of service by the kernel/toolchain/systemd/Caddy installation, denial of service by the trusted
trusted application, or secrets an application itself exposes. It does not application, or secrets already exposed through service configuration. v0.1
manage data migrations, databases, containers, Kubernetes, or application-specific does not manage data migrations, databases, containers, Kubernetes, or
activation. EQL therefore remains outside the generic deployment adapter until application-specific activation. Transfer remains an application-local,
its SQLite/catalog publication checkpoints can be modeled explicitly. reviewed step until a transport can be added without weakening host-key and
credential boundaries.
-98
View File
@@ -1,98 +0,0 @@
# Two-service build, approval, and deployment
This walkthrough assumes one small Linux host running systemd and Caddy, two Go
services, and a trusted Gitea build runner. The host has the same root-owned Tend
binary at `/usr/local/bin/tend`; it does not need Git or Go.
## Prepare the host once
1. Create `/etc/tend/services`, `/etc/tend/environment`, and
`/var/lib/tend/incoming`. The incoming and environment directories are
root-owned mode `0700`.
2. Install one schema-2 file per service, one per-service deployment-event log,
and one root-owned mode-`0600`
environment file per service. A singleton's shared environment file must
not define its configured listen key; its installed unit owns the live
address and Tend overrides only the transient candidate. Give each
singleton an imported Caddy handler and a root-owned handler template with
exactly one `{{UPSTREAM}}` marker; this is the bounded traffic handoff while
its fixed-address unit restarts.
3. Install the receive policy, forced `authorized_keys` entry, and exact sudoers
rule from `examples/server/` after replacing every placeholder. The sudoers
fragment preserves only `SSH_ORIGINAL_COMMAND`; the root receiver requires
its exact protocol value and refuses every other requested command.
4. Pin the server host key in a dedicated client file. Do not accept a new key
interactively during deployment.
5. Run `sudo tend check-server` and inspect the allowlisted service names.
6. Run `sudo tend reconcile --config /etc/tend/services/example-site.json
--json` before the first maintenance release. A settled report is expected;
any retained candidate or unknown handler must be understood before traffic
changes. The command is read-only.
## Build and approve
Trusted CI checks a clean pushed commit, uses a pinned Go toolchain, packages
twice, and requires byte-identical archives. It publishes the archive,
`RELEASE.json`, `BUILDINFO.json`, SPDX SBOM, and SHA-256 evidence.
The maintainer reads the candidate report and copies the exact approved digest
into the deployment command. Tend refuses a digest that is merely inferred from
the local file or differs from the produced value.
Inspect the exact downloaded bytes before transfer. This is a complete,
read-only artifact validation and does not stage a candidate:
```text
tend inspect --config /review/example-site.json \
--artifact /approved/example-site.tar.gz \
--sha256 <produced> --approve-sha256 <reviewed>
```
On the host, compare Tend's journal with the conventional service state before
and after activation. Reconciliation reports facts and never stops a unit,
rewrites a pointer, or changes Caddy:
```text
sudo tend reconcile --config /etc/tend/services/example-site.json
```
```text
tend push --target tend-deploy@server.example \
--known-hosts /secure/tend_known_hosts \
--identity /secure/tend_deploy_ed25519 \
--service example-site \
--artifact /approved/example-site.tar.gz \
--sha256 <produced> --approve-sha256 <reviewed> --activate
```
Repeat independently with `--service docs-site`. Both builds and transfers can
run concurrently. The host-wide lock serializes only activation because both
services share Caddy. Tend does not stop the other application.
## Failure and recovery exercises
- Change a candidate marker: activation must fail, preserve the active release,
and record the failed attempt without calling the candidate active.
- Make a Caddy template invalid: validation must fail and restore prior bytes.
- Make the public marker fail after an initially successful request: the
activation window must catch the transient routed failure and restore the
former slot/pointer.
- For a singleton, verify repeated canonical-origin requests remain successful
while Tend routes to the candidate, restarts the fixed-address unit, and
returns traffic to it. Inject failure at both Caddy reloads and require the
prior handler and pointer to be restored.
- Stop the previous blue/green slot during the activation window: Tend must
restore the old Caddy handler instead of accepting reduced continuity.
- Run `tend rollback --activate` for one service and verify the other service's
units, pointers, and public origin did not change.
- Interrupt a transfer: no release becomes active and the incomplete incoming
file is removed when the receiver exits.
- Interrupt singleton recovery after the candidate is proven. `tend reconcile
--json` must name the operation-scoped candidate unit, report whether it is
active and whether the handler file targets it, and perform no stop, restart,
reload, pointer, or state mutation.
After the soak, prune per service. Active and previous releases remain protected.
If activation fails, run `reconcile` before manual recovery so the retained
candidate, route, release pointers, and journal disagreement are preserved in
one bounded report.
+6 -28
View File
@@ -3,39 +3,17 @@
This subtree is licensed 0BSD so an operator can copy and adapt it without This subtree is licensed 0BSD so an operator can copy and adapt it without
bringing the Tend program's AGPL license into an application configuration. bringing the Tend program's AGPL license into an application configuration.
The blue/green example expects one separately reviewed environment file that The blue/green example expects separately reviewed environment files:
the two installed slots and transient validation use consistently:
```text ```text
# /etc/tend/environment/example-site.env # /etc/example-site/blue.env
APP_SECRET=replace-on-server EXAMPLE_LISTEN=127.0.0.1:8090
# /etc/example-site/green.env
EXAMPLE_LISTEN=127.0.0.1:8091
``` ```
The blue/green systemd slot units then read the nonsecret listen address from
`/etc/tend/slots/example-site-blue.env` or `-green.env`; Tend overrides only
the isolated candidate address. Secret values never enter `tend.json`.
The singleton example follows the same split: its shared root-only environment
file omits the configured listen key, the installed unit owns the live address,
and Tend supplies only the transient candidate address. This prevents a shared
environment file from overriding the isolated candidate port. Its imported
Caddy handler is also managed from one reviewed template. Tend temporarily
routes the canonical origin to the proven candidate while the fixed-address
unit restarts, then returns traffic to that unit only after it passes its local
checks.
Production configuration belongs outside the source checkout, owned by root, Production configuration belongs outside the source checkout, owned by root,
and not group- or world-writable. The Caddy handler template is an entire and not group- or world-writable. The Caddy handler template is an entire
imported handler fragment; the enclosing site, matchers, and routing precedence imported handler fragment; the enclosing site, matchers, and routing precedence
remain operator-owned. remain operator-owned.
`event_log` is a per-service, root-owned JSONL evidence stream below that
service's release root. Grant an Observatory agent read access explicitly; do
not make the release root broadly readable. `activation_window_seconds` keeps
canonical routed probes active after Caddy reload and keeps the previous
blue/green slot—or the singleton handoff candidate—under health/readiness
observation until the activation is recorded.
`server/` demonstrates the schema-2 receive policy, forced OpenSSH command,
restricted sudo entry, two independent service configurations, and secret-file
placement. The values are placeholders, not an installation script.
+1 -2
View File
@@ -6,8 +6,7 @@ After=network.target
Type=simple Type=simple
DynamicUser=yes DynamicUser=yes
ExecStart=/opt/example-site/slots/%i/example-site ExecStart=/opt/example-site/slots/%i/example-site
EnvironmentFile=/etc/tend/environment/example-site.env EnvironmentFile=/etc/example-site/%i.env
EnvironmentFile=/etc/tend/slots/example-site-%i.env
NoNewPrivileges=yes NoNewPrivileges=yes
PrivateTmp=yes PrivateTmp=yes
ProtectSystem=strict ProtectSystem=strict
+3 -6
View File
@@ -1,19 +1,16 @@
{ {
"schema_version": 2, "schema_version": 1,
"service": { "name": "example-site", "allowed_host": "example.test", "environment_file": "/etc/tend/environment/example-site.env" }, "service": { "name": "example-site", "allowed_host": "example.test" },
"build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" }, "build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" },
"deployment": { "deployment": {
"strategy": "blue_green", "strategy": "blue_green",
"root": "/opt/example-site", "root": "/opt/example-site",
"lock_file": "/run/lock/tend-deploy.lock", "lock_file": "/run/lock/gamertan-deploy.lock",
"state_file": "/opt/example-site/tend-state.json", "state_file": "/opt/example-site/tend-state.json",
"event_log": "/opt/example-site/deployment-events.jsonl",
"health_path": "/healthz", "health_path": "/healthz",
"readiness_path": "/readyz", "readiness_path": "/readyz",
"candidate_timeout_seconds": 30, "candidate_timeout_seconds": 30,
"activation_window_seconds": 10,
"smoke": [{ "path": "/", "contains": "Example site" }], "smoke": [{ "path": "/", "contains": "Example site" }],
"public_smoke": [{ "url": "https://example.test/", "contains": "Example site" }],
"blue_green": { "blue_green": {
"caddy_config": "/etc/caddy/Caddyfile", "caddy_config": "/etc/caddy/Caddyfile",
"caddy_handler": "/etc/caddy/example-site-handler.caddy", "caddy_handler": "/etc/caddy/example-site-handler.caddy",
-3
View File
@@ -1,3 +0,0 @@
# SPDX-License-Identifier: 0BSD
APP_MODE=development
APP_SECRET=replace-with-a-local-random-value
-2
View File
@@ -1,2 +0,0 @@
# SPDX-License-Identifier: 0BSD
restrict,command="sudo -n /usr/local/bin/tend receive --policy /etc/tend/receive-policy.json" ssh-ed25519 REPLACE_WITH_DEPLOY_KEY tend-deploy
-2
View File
@@ -1,2 +0,0 @@
# Managed by Tend. The enclosing site and route matchers remain operator-owned.
reverse_proxy {{UPSTREAM}}
@@ -1,2 +0,0 @@
# Managed by Tend. The enclosing site and route matchers remain operator-owned.
reverse_proxy {{UPSTREAM}}
@@ -1,3 +0,0 @@
# SPDX-License-Identifier: 0BSD
DOCS_LISTEN=127.0.0.1:8102
APP_SECRET=replace-on-server
@@ -1,2 +0,0 @@
# SPDX-License-Identifier: 0BSD
APP_SECRET=replace-on-server
-19
View File
@@ -1,19 +0,0 @@
# SPDX-License-Identifier: 0BSD
[Unit]
Description=Example singleton site
After=network.target
[Service]
Type=simple
DynamicUser=yes
ExecStart=/opt/example-site/current/example-site
EnvironmentFile=/etc/tend/environment/example-site.env
Environment=EXAMPLE_LISTEN=127.0.0.1:8092
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=strict
ProtectHome=yes
Restart=on-failure
[Install]
WantedBy=multi-user.target
-16
View File
@@ -1,16 +0,0 @@
{
"schema_version": 1,
"config_root": "/etc/tend/services",
"incoming_root": "/var/lib/tend/incoming",
"shared_lock_file": "/run/lock/tend-deploy.lock",
"services": {
"docs-site": {
"config": "/etc/tend/services/docs-site.json",
"max_artifact_bytes": 134217728
},
"example-site": {
"config": "/etc/tend/services/example-site.json",
"max_artifact_bytes": 134217728
}
}
}
-29
View File
@@ -1,29 +0,0 @@
{
"schema_version": 2,
"service": { "name": "docs-site", "allowed_host": "docs.example.test", "environment_file": "/etc/tend/environment/docs-site.env" },
"build": { "package": "./cmd/docs", "binary": "docs-site", "branch": "main" },
"deployment": {
"strategy": "singleton_candidate",
"root": "/opt/docs-site",
"lock_file": "/run/lock/tend-deploy.lock",
"state_file": "/opt/docs-site/tend-state.json",
"event_log": "/opt/docs-site/deployment-events.jsonl",
"health_path": "/healthz",
"readiness_path": "/readyz",
"candidate_timeout_seconds": 30,
"activation_window_seconds": 10,
"smoke": [{ "path": "/", "contains": "Documentation" }],
"public_smoke": [{ "url": "https://docs.example.test/", "contains": "Documentation" }],
"singleton": {
"unit": "docs-site.service",
"address": "127.0.0.1:8102",
"candidate_address": "127.0.0.1:18102",
"listen_env": "DOCS_LISTEN",
"current_link": "/opt/docs-site/current",
"previous_link": "/opt/docs-site/previous",
"caddy_config": "/etc/caddy/Caddyfile",
"caddy_handler": "/etc/caddy/docs-site-handler.caddy",
"caddy_handler_template": "/etc/tend/caddy/docs-site.template"
}
}
}
@@ -1,29 +0,0 @@
{
"schema_version": 2,
"service": { "name": "example-site", "allowed_host": "example.test", "environment_file": "/etc/tend/environment/example-site.env" },
"build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" },
"deployment": {
"strategy": "singleton_candidate",
"root": "/opt/example-site",
"lock_file": "/run/lock/tend-deploy.lock",
"state_file": "/opt/example-site/tend-state.json",
"event_log": "/opt/example-site/deployment-events.jsonl",
"health_path": "/healthz",
"readiness_path": "/readyz",
"candidate_timeout_seconds": 30,
"activation_window_seconds": 10,
"smoke": [{ "path": "/", "contains": "Example site" }],
"public_smoke": [{ "url": "https://example.test/", "contains": "Example site" }],
"singleton": {
"unit": "example-site.service",
"address": "127.0.0.1:8092",
"candidate_address": "127.0.0.1:18092",
"listen_env": "EXAMPLE_LISTEN",
"current_link": "/opt/example-site/current",
"previous_link": "/opt/example-site/previous",
"caddy_config": "/etc/caddy/Caddyfile",
"caddy_handler": "/etc/caddy/example-site-handler.caddy",
"caddy_handler_template": "/etc/tend/caddy/example-site.template"
}
}
}
@@ -1,2 +0,0 @@
# SPDX-License-Identifier: 0BSD
EXAMPLE_LISTEN=127.0.0.1:8090
@@ -1,2 +0,0 @@
# SPDX-License-Identifier: 0BSD
EXAMPLE_LISTEN=127.0.0.1:8091
-3
View File
@@ -1,3 +0,0 @@
# SPDX-License-Identifier: 0BSD
Defaults:tend-deploy env_keep += "SSH_ORIGINAL_COMMAND"
tend-deploy ALL=(root) NOPASSWD: /usr/local/bin/tend receive --policy /etc/tend/receive-policy.json
@@ -1,2 +0,0 @@
# Managed by Tend. The enclosing site and route matchers remain operator-owned.
reverse_proxy {{UPSTREAM}}
+5 -10
View File
@@ -1,29 +1,24 @@
{ {
"schema_version": 2, "schema_version": 1,
"service": { "name": "example-site", "allowed_host": "example.test", "environment_file": "/etc/tend/environment/example-site.env" }, "service": { "name": "example-site", "allowed_host": "example.test" },
"build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" }, "build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" },
"deployment": { "deployment": {
"strategy": "singleton_candidate", "strategy": "singleton_candidate",
"root": "/opt/example-site", "root": "/opt/example-site",
"lock_file": "/run/lock/tend-deploy.lock", "lock_file": "/run/lock/gamertan-deploy.lock",
"state_file": "/opt/example-site/tend-state.json", "state_file": "/opt/example-site/tend-state.json",
"event_log": "/opt/example-site/deployment-events.jsonl",
"health_path": "/healthz", "health_path": "/healthz",
"readiness_path": "/readyz", "readiness_path": "/readyz",
"candidate_timeout_seconds": 30, "candidate_timeout_seconds": 30,
"activation_window_seconds": 10,
"smoke": [{ "path": "/", "contains": "Example site" }], "smoke": [{ "path": "/", "contains": "Example site" }],
"public_smoke": [{ "url": "https://example.test/", "contains": "Example site" }],
"singleton": { "singleton": {
"unit": "example-site.service", "unit": "example-site.service",
"address": "127.0.0.1:8092", "address": "127.0.0.1:8092",
"candidate_address": "127.0.0.1:18092", "candidate_address": "127.0.0.1:18092",
"listen_env": "EXAMPLE_LISTEN", "listen_env": "EXAMPLE_LISTEN",
"environment": {},
"current_link": "/opt/example-site/current", "current_link": "/opt/example-site/current",
"previous_link": "/opt/example-site/previous", "previous_link": "/opt/example-site/previous"
"caddy_config": "/etc/caddy/Caddyfile",
"caddy_handler": "/etc/caddy/example-site-handler.caddy",
"caddy_handler_template": "/etc/tend/caddy/example-site.template"
} }
} }
} }
+5 -47
View File
@@ -9,17 +9,13 @@ import (
"fmt" "fmt"
"io" "io"
"net/netip" "net/netip"
"net/url"
"os" "os"
"path/filepath" "path/filepath"
"regexp" "regexp"
"strings" "strings"
) )
const ( const SchemaVersion = 1
SchemaVersion = 2
SharedLockFile = "/run/lock/tend-deploy.lock"
)
var ( var (
namePattern = regexp.MustCompile(`^[a-z][a-z0-9-]{1,62}$`) namePattern = regexp.MustCompile(`^[a-z][a-z0-9-]{1,62}$`)
@@ -39,7 +35,6 @@ type Config struct {
type Service struct { type Service struct {
Name string `json:"name"` Name string `json:"name"`
AllowedHost string `json:"allowed_host"` AllowedHost string `json:"allowed_host"`
EnvironmentFile string `json:"environment_file"`
} }
type Build struct { type Build struct {
@@ -56,13 +51,10 @@ type Deployment struct {
Root string `json:"root"` Root string `json:"root"`
LockFile string `json:"lock_file"` LockFile string `json:"lock_file"`
StateFile string `json:"state_file"` StateFile string `json:"state_file"`
EventLog string `json:"event_log"`
HealthPath string `json:"health_path"` HealthPath string `json:"health_path"`
ReadinessPath string `json:"readiness_path"` ReadinessPath string `json:"readiness_path"`
CandidateTimeoutSecs int `json:"candidate_timeout_seconds"` CandidateTimeoutSecs int `json:"candidate_timeout_seconds"`
ActivationWindowSecs int `json:"activation_window_seconds"`
Smoke []Smoke `json:"smoke"` Smoke []Smoke `json:"smoke"`
PublicSmoke []PublicSmoke `json:"public_smoke"`
BlueGreen *BlueGreen `json:"blue_green,omitempty"` BlueGreen *BlueGreen `json:"blue_green,omitempty"`
Singleton *Singleton `json:"singleton,omitempty"` Singleton *Singleton `json:"singleton,omitempty"`
} }
@@ -72,11 +64,6 @@ type Smoke struct {
Contains string `json:"contains"` Contains string `json:"contains"`
} }
type PublicSmoke struct {
URL string `json:"url"`
Contains string `json:"contains"`
}
type BlueGreen struct { type BlueGreen struct {
CaddyConfig string `json:"caddy_config"` CaddyConfig string `json:"caddy_config"`
CaddyHandler string `json:"caddy_handler"` CaddyHandler string `json:"caddy_handler"`
@@ -97,11 +84,9 @@ type Singleton struct {
Address string `json:"address"` Address string `json:"address"`
CandidateAddress string `json:"candidate_address"` CandidateAddress string `json:"candidate_address"`
ListenEnv string `json:"listen_env"` ListenEnv string `json:"listen_env"`
Environment map[string]string `json:"environment,omitempty"`
CurrentLink string `json:"current_link"` CurrentLink string `json:"current_link"`
PreviousLink string `json:"previous_link"` PreviousLink string `json:"previous_link"`
CaddyConfig string `json:"caddy_config"`
CaddyHandler string `json:"caddy_handler"`
CaddyHandlerTemplate string `json:"caddy_handler_template"`
} }
func Load(path string) (Config, error) { func Load(path string) (Config, error) {
@@ -155,12 +140,6 @@ func (c Config) Validate() error {
if c.Service.AllowedHost == "" || strings.ContainsAny(c.Service.AllowedHost, "/\\\x00\r\n\t ") { if c.Service.AllowedHost == "" || strings.ContainsAny(c.Service.AllowedHost, "/\\\x00\r\n\t ") {
return errors.New("service.allowed_host is invalid") return errors.New("service.allowed_host is invalid")
} }
if err := safeAbsolute("service.environment_file", c.Service.EnvironmentFile); err != nil {
return err
}
if !within("/etc/tend/environment", c.Service.EnvironmentFile) {
return errors.New("service.environment_file must be below /etc/tend/environment")
}
if !packagePattern.MatchString(c.Build.Package) || strings.Contains(c.Build.Package, "..") { if !packagePattern.MatchString(c.Build.Package) || strings.Contains(c.Build.Package, "..") {
return errors.New("build.package must be a local package without traversal") return errors.New("build.package must be a local package without traversal")
} }
@@ -192,21 +171,12 @@ func (c Config) Validate() error {
if filepath.Clean(d.StateFile) == filepath.Clean(d.Root) || !within(d.Root, d.StateFile) { if filepath.Clean(d.StateFile) == filepath.Clean(d.Root) || !within(d.Root, d.StateFile) {
return errors.New("deployment.state_file must be below deployment.root") return errors.New("deployment.state_file must be below deployment.root")
} }
if err := safeAbsolute("deployment.event_log", d.EventLog); err != nil {
return err
}
if filepath.Clean(d.EventLog) == filepath.Clean(d.Root) || !within(d.Root, d.EventLog) || filepath.Clean(d.EventLog) == filepath.Clean(d.StateFile) {
return errors.New("deployment.event_log must be a distinct file below deployment.root")
}
if !safeHTTPPath(d.HealthPath) || !safeHTTPPath(d.ReadinessPath) { if !safeHTTPPath(d.HealthPath) || !safeHTTPPath(d.ReadinessPath) {
return errors.New("health and readiness paths must be absolute HTTP paths") return errors.New("health and readiness paths must be absolute HTTP paths")
} }
if d.CandidateTimeoutSecs < 2 || d.CandidateTimeoutSecs > 300 { if d.CandidateTimeoutSecs < 2 || d.CandidateTimeoutSecs > 300 {
return errors.New("candidate_timeout_seconds must be between 2 and 300") return errors.New("candidate_timeout_seconds must be between 2 and 300")
} }
if d.ActivationWindowSecs < 1 || d.ActivationWindowSecs > 120 {
return errors.New("activation_window_seconds must be between 1 and 120")
}
if len(d.Smoke) == 0 || len(d.Smoke) > 32 { if len(d.Smoke) == 0 || len(d.Smoke) > 32 {
return errors.New("deployment.smoke must contain 1 to 32 checks") return errors.New("deployment.smoke must contain 1 to 32 checks")
} }
@@ -215,15 +185,6 @@ func (c Config) Validate() error {
return fmt.Errorf("deployment.smoke[%d] is invalid", i) return fmt.Errorf("deployment.smoke[%d] is invalid", i)
} }
} }
if len(d.PublicSmoke) == 0 || len(d.PublicSmoke) > 16 {
return errors.New("deployment.public_smoke must contain 1 to 16 checks")
}
for i, smoke := range d.PublicSmoke {
parsed, err := url.Parse(smoke.URL)
if err != nil || parsed.Scheme != "https" || parsed.Host == "" || parsed.User != nil || parsed.Fragment != "" || parsed.RawQuery != "" || parsed.Opaque != "" || smoke.Contains == "" || len(smoke.Contains) > 4096 || strings.ContainsRune(smoke.Contains, '\x00') {
return fmt.Errorf("deployment.public_smoke[%d] is invalid", i)
}
}
switch d.Strategy { switch d.Strategy {
case "blue_green": case "blue_green":
if d.BlueGreen == nil || d.Singleton != nil { if d.BlueGreen == nil || d.Singleton != nil {
@@ -312,14 +273,11 @@ func validateSingleton(root string, s Singleton) error {
if s.CurrentLink == s.PreviousLink { if s.CurrentLink == s.PreviousLink {
return errors.New("current and previous links must differ") return errors.New("current and previous links must differ")
} }
for label, path := range map[string]string{"caddy_config": s.CaddyConfig, "caddy_handler": s.CaddyHandler, "caddy_handler_template": s.CaddyHandlerTemplate} { for key, value := range s.Environment {
if err := safeAbsolute("deployment.singleton."+label, path); err != nil { if !regexp.MustCompile(`^[A-Z][A-Z0-9_]{0,63}$`).MatchString(key) || strings.ContainsAny(value, "\x00\r\n") {
return err return fmt.Errorf("singleton environment entry %q is invalid", key)
} }
} }
if filepath.Clean(s.CaddyHandler) == filepath.Clean(s.CaddyHandlerTemplate) {
return errors.New("singleton Caddy handler and template must be different files")
}
return nil return nil
} }
+5 -27
View File
@@ -9,14 +9,13 @@ import (
func validConfig() Config { func validConfig() Config {
return Config{ return Config{
SchemaVersion: 2, SchemaVersion: 1,
Service: Service{Name: "example-site", AllowedHost: "example.test", EnvironmentFile: "/etc/tend/environment/example-site.env"}, Service: Service{Name: "example-site", AllowedHost: "example.test"},
Build: Build{Package: "./cmd/site", Binary: "example-site", Branch: "main"}, Build: Build{Package: "./cmd/site", Binary: "example-site", Branch: "main"},
Deployment: Deployment{ Deployment: Deployment{
Strategy: "blue_green", Root: "/opt/example-site", LockFile: SharedLockFile, Strategy: "blue_green", Root: "/opt/example-site", LockFile: "/run/lock/example-site.lock",
StateFile: "/opt/example-site/state.json", EventLog: "/opt/example-site/deployment-events.jsonl", HealthPath: "/healthz", ReadinessPath: "/readyz", StateFile: "/opt/example-site/state.json", HealthPath: "/healthz", ReadinessPath: "/readyz",
CandidateTimeoutSecs: 30, ActivationWindowSecs: 10, Smoke: []Smoke{{Path: "/", Contains: "Example"}}, CandidateTimeoutSecs: 30, Smoke: []Smoke{{Path: "/", Contains: "Example"}},
PublicSmoke: []PublicSmoke{{URL: "https://example.test/", Contains: "Example"}},
BlueGreen: &BlueGreen{ BlueGreen: &BlueGreen{
CaddyConfig: "/etc/caddy/Caddyfile", CaddyHandler: "/etc/caddy/example.caddy", CaddyConfig: "/etc/caddy/Caddyfile", CaddyHandler: "/etc/caddy/example.caddy",
CaddyHandlerTemplate: "/etc/example/caddy.template", CaddyHandlerTemplate: "/etc/example/caddy.template",
@@ -34,24 +33,6 @@ func TestValidateAcceptsBlueGreen(t *testing.T) {
} }
} }
func TestValidateSingletonRequiresDistinctCaddyHandoffFiles(t *testing.T) {
cfg := validConfig()
cfg.Deployment.Strategy = "singleton_candidate"
cfg.Deployment.BlueGreen = nil
cfg.Deployment.Singleton = &Singleton{
Unit: "example-site.service", Address: "127.0.0.1:8092", CandidateAddress: "127.0.0.1:18092", ListenEnv: "EXAMPLE_LISTEN",
CurrentLink: "/opt/example-site/current", PreviousLink: "/opt/example-site/previous", CaddyConfig: "/etc/caddy/Caddyfile",
CaddyHandler: "/etc/caddy/example-site.caddy", CaddyHandlerTemplate: "/etc/tend/caddy/example-site.template",
}
if err := cfg.Validate(); err != nil {
t.Fatal(err)
}
cfg.Deployment.Singleton.CaddyHandlerTemplate = cfg.Deployment.Singleton.CaddyHandler
if err := cfg.Validate(); err == nil {
t.Fatal("expected shared handler/template path to be rejected")
}
}
func TestValidateRejectsHostileValues(t *testing.T) { func TestValidateRejectsHostileValues(t *testing.T) {
tests := map[string]func(*Config){ tests := map[string]func(*Config){
"unknown strategy": func(c *Config) { c.Deployment.Strategy = "shell" }, "unknown strategy": func(c *Config) { c.Deployment.Strategy = "shell" },
@@ -60,9 +41,6 @@ func TestValidateRejectsHostileValues(t *testing.T) {
"traversal": func(c *Config) { c.Build.Package = "./cmd/../secret" }, "traversal": func(c *Config) { c.Build.Package = "./cmd/../secret" },
"shared slot": func(c *Config) { c.Deployment.BlueGreen.Green.Link = c.Deployment.BlueGreen.Blue.Link }, "shared slot": func(c *Config) { c.Deployment.BlueGreen.Green.Link = c.Deployment.BlueGreen.Blue.Link },
"bad smoke": func(c *Config) { c.Deployment.Smoke[0].Path = "https://attacker.test/" }, "bad smoke": func(c *Config) { c.Deployment.Smoke[0].Path = "https://attacker.test/" },
"bad public smoke": func(c *Config) { c.Deployment.PublicSmoke[0].URL = "http://example.test/" },
"public secret query": func(c *Config) { c.Deployment.PublicSmoke[0].URL = "https://example.test/?token=secret" },
"environment sibling": func(c *Config) { c.Service.EnvironmentFile = "/etc/tend/environment-old/example.env" },
} }
for name, mutate := range tests { for name, mutate := range tests {
t.Run(name, func(t *testing.T) { t.Run(name, func(t *testing.T) {
+82 -486
View File
@@ -14,7 +14,6 @@ import (
"time" "time"
"gamertan.com/tend/internal/config" "gamertan.com/tend/internal/config"
"gamertan.com/tend/internal/eventlog"
"gamertan.com/tend/internal/state" "gamertan.com/tend/internal/state"
) )
@@ -30,8 +29,6 @@ type Report struct {
Release string `json:"release,omitempty"` Release string `json:"release,omitempty"`
ActiveRelease string `json:"active_release,omitempty"` ActiveRelease string `json:"active_release,omitempty"`
PreviousRelease string `json:"previous_release,omitempty"` PreviousRelease string `json:"previous_release,omitempty"`
EventWarnings int `json:"event_warnings,omitempty"`
LeaseCleanupPending bool `json:"lease_cleanup_pending,omitempty"`
} }
type Status struct { type Status struct {
State *state.Record `json:"state,omitempty"` State *state.Record `json:"state,omitempty"`
@@ -39,24 +36,15 @@ type Status struct {
StateInitialized bool `json:"state_initialized"` StateInitialized bool `json:"state_initialized"`
} }
type retainedCandidateError struct{ cause error }
func (e *retainedCandidateError) Error() string { return e.cause.Error() }
func (e *retainedCandidateError) Unwrap() error { return e.cause }
type Manager struct { type Manager struct {
Operator Operator Operator Operator
Now func() time.Time Now func() time.Time
Prepare func(config.Config, string, string, string) (string, error) Prepare func(config.Config, string, string, string) (string, error)
Inspect func(config.Config, string, string, string) error Inspect func(config.Config, string, string, string) error
ReadIdentity func(string) (releaseIdentity, error)
OperationID func() (string, error)
AppendEvent func(string, eventlog.Event) error
Sleep func(context.Context, time.Duration) error
} }
func NewManager(operator Operator) Manager { func NewManager(operator Operator) Manager {
return Manager{Operator: operator, Now: time.Now, Prepare: prepareRelease, Inspect: inspectArtifact, ReadIdentity: readReleaseIdentity, OperationID: eventlog.OperationID, AppendEvent: eventlog.Append, Sleep: sleepContext} return Manager{Operator: operator, Now: time.Now, Prepare: prepareRelease, Inspect: inspectArtifact}
} }
func (m Manager) Deploy(ctx context.Context, cfg config.Config, request Request) (Report, error) { func (m Manager) Deploy(ctx context.Context, cfg config.Config, request Request) (Report, error) {
@@ -78,186 +66,26 @@ func (m Manager) Deploy(ctx context.Context, cfg config.Config, request Request)
if err != nil { if err != nil {
return Report{}, err return Report{}, err
} }
started := m.Now()
eventWarnings := 0
identity, identityErr := m.ReadIdentity(release)
if identityErr != nil {
eventWarnings++
}
operationID := ""
if m.OperationID != nil {
operationID, err = m.OperationID()
if err != nil {
eventWarnings++
operationID = ""
}
}
if cfg.Deployment.Strategy == "singleton_candidate" && operationID == "" {
return Report{}, errors.New("singleton activation requires a fresh operation identity")
}
emit := func(phase, slot, outcome string) {
if m.AppendEvent == nil || identityErr != nil || operationID == "" {
return
}
event := eventlog.Event{Version: eventlog.Version, OperationID: operationID, Service: cfg.Service.Name, ArtifactDigest: request.ApprovedSHA256, Commit: identity.Commit, ReleaseVersion: identity.Version, Phase: phase, Slot: slot, DurationMillis: max(0, m.Now().Sub(started).Milliseconds()), Outcome: outcome, ObservedAt: m.Now().UTC().Format(time.RFC3339Nano)}
if eventErr := m.AppendEvent(cfg.Deployment.EventLog, event); eventErr != nil {
eventWarnings++
}
}
record, err := loadOrBootstrap(cfg, m.Now()) record, err := loadOrBootstrap(cfg, m.Now())
if err != nil { if err != nil {
return Report{}, err return Report{}, err
} }
leasePath := state.CandidateLeasePath(cfg.Deployment.StateFile)
var candidateLease state.CandidateLease
if cfg.Deployment.Strategy == "singleton_candidate" {
_, exists, leaseErr := loadCandidateLease(cfg)
if leaseErr != nil {
return Report{}, leaseErr
}
if record.CandidateRelease != "" || exists {
return Report{}, errors.New("singleton candidate lease is unresolved; run tend reconcile --json before another activation")
}
}
attemptAt := m.Now().UTC().Format(time.RFC3339)
record.DesiredRelease = release
record.CandidateRelease = release
record.LastAttemptRelease = release
record.LastAttemptOutcome = "running"
record.LastAttemptAt = attemptAt
record.UpdatedAt = attemptAt
if cfg.Deployment.Strategy == "singleton_candidate" {
candidateUnit, unitErr := singletonCandidateUnit(cfg.Service.Name, operationID)
if unitErr != nil {
return Report{}, unitErr
}
candidateLease = state.CandidateLease{SchemaVersion: state.CandidateLeaseSchemaVersion, Service: cfg.Service.Name, OperationID: operationID, Release: release, Unit: candidateUnit, Address: cfg.Deployment.Singleton.CandidateAddress, StartedAt: attemptAt}
}
if err := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, record); err != nil {
return Report{}, err
}
if cfg.Deployment.Strategy == "singleton_candidate" {
if err := state.StoreCandidateLease(leasePath, cfg.Deployment.Root, candidateLease); err != nil {
failed := record
clearCandidateLease(&failed)
failed.LastAttemptOutcome = "failed"
failed.UpdatedAt = m.Now().UTC().Format(time.RFC3339)
if storeErr := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, failed); storeErr != nil {
return Report{}, errors.Join(err, storeErr)
}
return Report{}, err
}
}
emit("candidate", inactiveSlot(cfg, record), "running")
switch cfg.Deployment.Strategy { switch cfg.Deployment.Strategy {
case "blue_green": case "blue_green":
err = m.deployBlueGreen(ctx, cfg, record, release) err = m.deployBlueGreen(ctx, cfg, record, release)
case "singleton_candidate": case "singleton_candidate":
err = m.deploySingleton(ctx, cfg, record, release, candidateLease.Unit) err = m.deploySingleton(ctx, cfg, record, release)
default: default:
err = errors.New("unsupported strategy") err = errors.New("unsupported strategy")
} }
if err != nil { if err != nil {
failed := record return Report{}, err
var retained *retainedCandidateError
if !errors.As(err, &retained) {
if cleanupErr := state.RemoveCandidateLease(leasePath); cleanupErr != nil {
err = errors.Join(err, fmt.Errorf("candidate lease cleanup failed: %w", cleanupErr))
} else {
clearCandidateLease(&failed)
} }
}
failed.LastAttemptOutcome = "failed"
failed.UpdatedAt = m.Now().UTC().Format(time.RFC3339)
if storeErr := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, failed); storeErr != nil {
err = errors.Join(err, storeErr)
}
emit("activation", inactiveSlot(cfg, record), "failed")
return Report{EventWarnings: eventWarnings}, err
}
emit("activation", inactiveSlot(cfg, record), "succeeded")
updated, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy) updated, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
if err != nil { if err != nil {
return Report{}, err return Report{}, err
} }
report := Report{Validated: true, Mutation: "activated", Release: release, ActiveRelease: updated.ActiveRelease, PreviousRelease: updated.PreviousRelease, EventWarnings: eventWarnings} return Report{Validated: true, Mutation: "activated", Release: release, ActiveRelease: updated.ActiveRelease, PreviousRelease: updated.PreviousRelease}, nil
if cfg.Deployment.Strategy == "singleton_candidate" {
if cleanupErr := state.RemoveCandidateLease(leasePath); cleanupErr != nil {
report.LeaseCleanupPending = true
}
}
return report, nil
}
func singletonCandidateUnit(service, operationID string) (string, error) {
if len(operationID) != 32 {
return "", errors.New("candidate operation identity is invalid")
}
for _, character := range operationID {
if !strings.ContainsRune("0123456789abcdef", character) {
return "", errors.New("candidate operation identity is invalid")
}
}
return service + "-tend-candidate-" + operationID[:12] + ".service", nil
}
func clearCandidateLease(record *state.Record) {
record.CandidateRelease = ""
}
func loadCandidateLease(cfg config.Config) (state.CandidateLease, bool, error) {
lease, err := state.LoadCandidateLease(state.CandidateLeasePath(cfg.Deployment.StateFile), cfg.Deployment.Root, cfg.Service.Name)
if err == nil {
return lease, true, nil
}
if os.IsNotExist(err) {
return state.CandidateLease{}, false, nil
}
return state.CandidateLease{}, false, fmt.Errorf("load singleton candidate lease: %w", err)
}
type releaseIdentity struct {
Version string `json:"version"`
Commit string `json:"commit"`
}
func readReleaseIdentity(release string) (releaseIdentity, error) {
b, err := os.ReadFile(filepath.Join(release, "RELEASE.json"))
if err != nil {
return releaseIdentity{}, fmt.Errorf("read installed release identity: %w", err)
}
if len(b) > 1<<20 {
return releaseIdentity{}, errors.New("installed release identity is too large")
}
var identity releaseIdentity
if err := json.Unmarshal(b, &identity); err != nil {
return releaseIdentity{}, errors.New("decode installed release identity")
}
if identity.Version == "" || identity.Commit == "" {
return releaseIdentity{}, errors.New("installed release identity is incomplete")
}
return identity, nil
}
func inactiveSlot(cfg config.Config, record state.Record) string {
if cfg.Deployment.Strategy == "singleton_candidate" {
return "singleton"
}
if record.ActiveSlot == "blue" {
return "green"
}
return "blue"
}
func sleepContext(ctx context.Context, duration time.Duration) error {
timer := time.NewTimer(duration)
defer timer.Stop()
select {
case <-ctx.Done():
return ctx.Err()
case <-timer.C:
return nil
}
} }
func loadOrBootstrap(cfg config.Config, now time.Time) (state.Record, error) { func loadOrBootstrap(cfg config.Config, now time.Time) (state.Record, error) {
@@ -275,13 +103,13 @@ func loadOrBootstrap(cfg config.Config, now time.Time) (state.Record, error) {
if err != nil { if err != nil {
return state.Record{}, fmt.Errorf("bootstrap active slot: %w", err) return state.Record{}, fmt.Errorf("bootstrap active slot: %w", err)
} }
return state.Record{SchemaVersion: state.SchemaVersion, Strategy: cfg.Deployment.Strategy, DesiredRelease: release, ActiveSlot: slot, ActiveRelease: release, UpdatedAt: now.UTC().Format(time.RFC3339)}, nil return state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, ActiveSlot: slot, ActiveRelease: release, UpdatedAt: now.UTC().Format(time.RFC3339)}, nil
case "singleton_candidate": case "singleton_candidate":
release, err := resolveReleaseLink(cfg.Deployment.Root, cfg.Deployment.Singleton.CurrentLink) release, err := resolveReleaseLink(cfg.Deployment.Root, cfg.Deployment.Singleton.CurrentLink)
if err != nil { if err != nil {
return state.Record{}, fmt.Errorf("bootstrap singleton: %w", err) return state.Record{}, fmt.Errorf("bootstrap singleton: %w", err)
} }
return state.Record{SchemaVersion: state.SchemaVersion, Strategy: cfg.Deployment.Strategy, DesiredRelease: release, ActiveSlot: "singleton", ActiveRelease: release, UpdatedAt: now.UTC().Format(time.RFC3339)}, nil return state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, ActiveSlot: "singleton", ActiveRelease: release, UpdatedAt: now.UTC().Format(time.RFC3339)}, nil
} }
return state.Record{}, errors.New("unsupported strategy") return state.Record{}, errors.New("unsupported strategy")
} }
@@ -349,25 +177,68 @@ func (m Manager) deployBlueGreen(ctx context.Context, cfg config.Config, record
if err = m.probeAll(ctx, cfg, slot.Address); err != nil { if err = m.probeAll(ctx, cfg, slot.Address); err != nil {
return fmt.Errorf("post-activation smoke failed: %w", err) return fmt.Errorf("post-activation smoke failed: %w", err)
} }
if err = m.probePublic(ctx, cfg, true); err != nil { next := state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, ActiveSlot: inactive, ActiveRelease: release, PreviousSlot: record.ActiveSlot, PreviousRelease: record.ActiveRelease, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
return fmt.Errorf("public-origin smoke failed: %w", err)
}
previous := slotConfig(bg, record.ActiveSlot)
if err = m.continuityWindow(ctx, cfg, previous.Address, true); err != nil {
return fmt.Errorf("activation continuity failed: %w", err)
}
next := state.Record{SchemaVersion: state.SchemaVersion, Strategy: cfg.Deployment.Strategy, DesiredRelease: release, ActiveSlot: inactive, ActiveRelease: release, PreviousSlot: record.ActiveSlot, PreviousRelease: record.ActiveRelease, LastAttemptRelease: release, LastAttemptOutcome: "succeeded", LastAttemptAt: record.LastAttemptAt, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
if err = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next); err != nil { if err = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next); err != nil {
return err return err
} }
return nil return nil
} }
func (m Manager) deploySingleton(ctx context.Context, cfg config.Config, record state.Record, release, candidateUnit string) (err error) { func (m Manager) deploySingleton(ctx context.Context, cfg config.Config, record state.Record, release string) (err error) {
if err = m.activateSingletonRelease(ctx, cfg, release, candidateUnit, true); err != nil { single := *cfg.Deployment.Singleton
candidateUnit := cfg.Service.Name + "-tend-candidate.service"
env := copyMap(single.Environment)
env[single.ListenEnv] = single.CandidateAddress
binary := filepath.Join(release, cfg.Build.Binary)
if err = m.Operator.StartCandidate(ctx, candidateUnit, binary, env); err != nil {
return err return err
} }
next := state.Record{SchemaVersion: state.SchemaVersion, Strategy: cfg.Deployment.Strategy, DesiredRelease: release, ActiveSlot: "singleton", ActiveRelease: release, PreviousSlot: "singleton", PreviousRelease: record.ActiveRelease, LastAttemptRelease: release, LastAttemptOutcome: "succeeded", LastAttemptAt: record.LastAttemptAt, UpdatedAt: m.Now().UTC().Format(time.RFC3339)} defer func() {
stopCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_ = m.Operator.Stop(stopCtx, candidateUnit)
}()
if err = m.probeAll(ctx, cfg, single.CandidateAddress); err != nil {
return fmt.Errorf("candidate failed: %w", err)
}
oldCurrent, err := resolveReleaseLink(cfg.Deployment.Root, single.CurrentLink)
if err != nil {
return err
}
oldPrevious, previousErr := resolveReleaseLink(cfg.Deployment.Root, single.PreviousLink)
currentChanged := false
previousChanged := false
defer func() {
if err == nil {
return
}
if currentChanged {
_ = replaceSymlink(single.CurrentLink, oldCurrent)
_ = m.Operator.Restart(ctx, single.Unit)
}
if previousChanged {
if previousErr == nil {
_ = replaceSymlink(single.PreviousLink, oldPrevious)
} else {
_ = removeSymlink(single.PreviousLink)
}
}
}()
if err = replaceSymlink(single.PreviousLink, oldCurrent); err != nil {
return err
}
previousChanged = true
if err = replaceSymlink(single.CurrentLink, release); err != nil {
return err
}
currentChanged = true
if err = m.Operator.Restart(ctx, single.Unit); err != nil {
return err
}
if err = m.probeAll(ctx, cfg, single.Address); err != nil {
return fmt.Errorf("post-activation smoke failed: %w", err)
}
next := state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, ActiveSlot: "singleton", ActiveRelease: release, PreviousSlot: "singleton", PreviousRelease: record.ActiveRelease, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
if err = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next); err != nil { if err = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next); err != nil {
return err return err
} }
@@ -387,119 +258,18 @@ func (m Manager) Rollback(ctx context.Context, cfg config.Config) (state.Record,
if record.PreviousRelease == "" { if record.PreviousRelease == "" {
return state.Record{}, errors.New("no previous release is recorded") return state.Record{}, errors.New("no previous release is recorded")
} }
leasePath := state.CandidateLeasePath(cfg.Deployment.StateFile)
if cfg.Deployment.Strategy == "singleton_candidate" {
_, exists, leaseErr := loadCandidateLease(cfg)
if leaseErr != nil {
return state.Record{}, leaseErr
}
if record.CandidateRelease != "" || exists {
return state.Record{}, errors.New("singleton candidate lease is unresolved; run tend reconcile --json before rollback")
}
}
started := m.Now()
identity, identityErr := m.ReadIdentity(record.PreviousRelease)
digest, digestErr := releaseDigest(record.PreviousRelease)
operationID := ""
if m.OperationID != nil {
operationID, err = m.OperationID()
}
if cfg.Deployment.Strategy == "singleton_candidate" && (err != nil || operationID == "") {
return state.Record{}, errors.New("singleton rollback requires a fresh operation identity")
}
candidateUnit := ""
rollbackAttempt := record
if cfg.Deployment.Strategy == "singleton_candidate" {
candidateUnit, err = singletonCandidateUnit(cfg.Service.Name, operationID)
if err != nil {
return state.Record{}, err
}
attemptAt := m.Now().UTC().Format(time.RFC3339)
rollbackAttempt.DesiredRelease = record.PreviousRelease
rollbackAttempt.CandidateRelease = record.PreviousRelease
rollbackAttempt.LastAttemptRelease = record.PreviousRelease
rollbackAttempt.LastAttemptOutcome = "running"
rollbackAttempt.LastAttemptAt = attemptAt
rollbackAttempt.UpdatedAt = attemptAt
if err = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, rollbackAttempt); err != nil {
return state.Record{}, err
}
lease := state.CandidateLease{SchemaVersion: state.CandidateLeaseSchemaVersion, Service: cfg.Service.Name, OperationID: operationID, Release: record.PreviousRelease, Unit: candidateUnit, Address: cfg.Deployment.Singleton.CandidateAddress, StartedAt: attemptAt}
if err = state.StoreCandidateLease(leasePath, cfg.Deployment.Root, lease); err != nil {
failed := rollbackAttempt
clearCandidateLease(&failed)
failed.LastAttemptOutcome = "failed"
failed.UpdatedAt = m.Now().UTC().Format(time.RFC3339)
if storeErr := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, failed); storeErr != nil {
return state.Record{}, errors.Join(err, storeErr)
}
return state.Record{}, err
}
}
emit := func(outcome string) {
if m.AppendEvent == nil || identityErr != nil || digestErr != nil || operationID == "" {
return
}
event := eventlog.Event{Version: eventlog.Version, OperationID: operationID, Service: cfg.Service.Name, ArtifactDigest: digest, Commit: identity.Commit, ReleaseVersion: identity.Version, Phase: "rollback", Slot: record.PreviousSlot, DurationMillis: max(0, m.Now().Sub(started).Milliseconds()), Outcome: outcome, ObservedAt: m.Now().UTC().Format(time.RFC3339Nano)}
_ = m.AppendEvent(cfg.Deployment.EventLog, event)
}
emit("running")
switch cfg.Deployment.Strategy { switch cfg.Deployment.Strategy {
case "blue_green": case "blue_green":
err = m.rollbackBlueGreen(ctx, cfg, record) err = m.rollbackBlueGreen(ctx, cfg, record)
case "singleton_candidate": case "singleton_candidate":
err = m.rollbackSingleton(ctx, cfg, record, candidateUnit) err = m.rollbackSingleton(ctx, cfg, record)
default: default:
err = errors.New("unsupported strategy") err = errors.New("unsupported strategy")
} }
if err != nil { if err != nil {
if cfg.Deployment.Strategy == "singleton_candidate" {
failed := rollbackAttempt
var retained *retainedCandidateError
if !errors.As(err, &retained) {
if cleanupErr := state.RemoveCandidateLease(leasePath); cleanupErr != nil {
err = errors.Join(err, fmt.Errorf("candidate lease cleanup failed: %w", cleanupErr))
} else {
clearCandidateLease(&failed)
}
}
failed.LastAttemptOutcome = "failed"
failed.UpdatedAt = m.Now().UTC().Format(time.RFC3339)
if storeErr := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, failed); storeErr != nil {
err = errors.Join(err, storeErr)
}
}
emit("failed")
return state.Record{}, err return state.Record{}, err
} }
emit("succeeded") return state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
updated, loadErr := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
if loadErr != nil {
return state.Record{}, loadErr
}
if cfg.Deployment.Strategy == "singleton_candidate" {
if cleanupErr := state.RemoveCandidateLease(leasePath); cleanupErr != nil {
return updated, fmt.Errorf("rollback succeeded but candidate lease cleanup is pending; run tend reconcile --json: %w", cleanupErr)
}
}
return updated, nil
}
func releaseDigest(release string) (string, error) {
name := filepath.Base(release)
if !strings.HasPrefix(name, "sha256-") {
return "", errors.New("release is not content addressed")
}
digest := strings.TrimPrefix(name, "sha256-")
if len(digest) != 64 {
return "", errors.New("release digest is invalid")
}
for _, character := range digest {
if !strings.ContainsRune("0123456789abcdef", character) {
return "", errors.New("release digest is invalid")
}
}
return digest, nil
} }
func (m Manager) rollbackBlueGreen(ctx context.Context, cfg config.Config, record state.Record) (err error) { func (m Manager) rollbackBlueGreen(ctx context.Context, cfg config.Config, record state.Record) (err error) {
bg := *cfg.Deployment.BlueGreen bg := *cfg.Deployment.BlueGreen
@@ -543,150 +313,33 @@ func (m Manager) rollbackBlueGreen(ctx context.Context, cfg config.Config, recor
if err = m.probeHealthReadiness(ctx, cfg, slot.Address); err != nil { if err = m.probeHealthReadiness(ctx, cfg, slot.Address); err != nil {
return err return err
} }
if err = m.probePublic(ctx, cfg, false); err != nil { next := state.Record{SchemaVersion: 1, Strategy: record.Strategy, ActiveSlot: record.PreviousSlot, ActiveRelease: record.PreviousRelease, PreviousSlot: record.ActiveSlot, PreviousRelease: record.ActiveRelease, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
return err
}
next := state.Record{SchemaVersion: state.SchemaVersion, Strategy: record.Strategy, DesiredRelease: record.PreviousRelease, ActiveSlot: record.PreviousSlot, ActiveRelease: record.PreviousRelease, PreviousSlot: record.ActiveSlot, PreviousRelease: record.ActiveRelease, LastAttemptRelease: record.PreviousRelease, LastAttemptOutcome: "rolled_back", LastAttemptAt: m.Now().UTC().Format(time.RFC3339), UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
return state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next) return state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next)
} }
func (m Manager) rollbackSingleton(ctx context.Context, cfg config.Config, record state.Record, candidateUnit string) (err error) { func (m Manager) rollbackSingleton(ctx context.Context, cfg config.Config, record state.Record) (err error) {
if err = m.activateSingletonRelease(ctx, cfg, record.PreviousRelease, candidateUnit, false); err != nil {
return err
}
next := state.Record{SchemaVersion: state.SchemaVersion, Strategy: record.Strategy, DesiredRelease: record.PreviousRelease, ActiveSlot: "singleton", ActiveRelease: record.PreviousRelease, PreviousSlot: "singleton", PreviousRelease: record.ActiveRelease, LastAttemptRelease: record.PreviousRelease, LastAttemptOutcome: "rolled_back", LastAttemptAt: m.Now().UTC().Format(time.RFC3339), UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
return state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next)
}
// activateSingletonRelease keeps public traffic on a proven process while the
// installed fixed-address unit changes release. The transient candidate first
// receives traffic, remains healthy through the handoff, and is stopped only
// after Caddy points back to the verified installed unit.
func (m Manager) activateSingletonRelease(ctx context.Context, cfg config.Config, release, candidateUnit string, checkMarkers bool) (err error) {
single := *cfg.Deployment.Singleton single := *cfg.Deployment.Singleton
env := map[string]string{single.ListenEnv: single.CandidateAddress}
binary := filepath.Join(release, cfg.Build.Binary)
if err = m.Operator.StartCandidate(ctx, candidateUnit, binary, cfg.Service.EnvironmentFile, env); err != nil {
return err
}
stopCandidate := true
defer func() {
if !stopCandidate {
return
}
stopCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
_ = m.Operator.Stop(stopCtx, candidateUnit)
}()
probeLocal := m.probeHealthReadiness
if checkMarkers {
probeLocal = m.probeAll
}
if err = probeLocal(ctx, cfg, single.CandidateAddress); err != nil {
return fmt.Errorf("candidate failed: %w", err)
}
oldHandler, err := os.ReadFile(single.CaddyHandler)
if err != nil {
return fmt.Errorf("read current Caddy handler: %w", err)
}
oldCurrent, err := resolveReleaseLink(cfg.Deployment.Root, single.CurrentLink) oldCurrent, err := resolveReleaseLink(cfg.Deployment.Root, single.CurrentLink)
if err != nil { if err != nil {
return err return err
} }
oldPrevious, previousErr := resolveReleaseLink(cfg.Deployment.Root, single.PreviousLink) if err = replaceSymlink(single.CurrentLink, record.PreviousRelease); err != nil {
handlerChanged := false return err
currentChanged := false }
previousChanged := false
defer func() { defer func() {
if err == nil { if err != nil {
return _ = replaceSymlink(single.CurrentLink, oldCurrent)
} _ = m.Operator.Restart(ctx, single.Unit)
recoveryErr := error(nil)
if currentChanged {
if restoreErr := replaceSymlink(single.CurrentLink, oldCurrent); restoreErr != nil {
recoveryErr = errors.Join(recoveryErr, restoreErr)
} else if restoreErr = m.Operator.Restart(ctx, single.Unit); restoreErr != nil {
recoveryErr = errors.Join(recoveryErr, restoreErr)
} else if restoreErr = m.probeHealthReadiness(ctx, cfg, single.Address); restoreErr != nil {
recoveryErr = errors.Join(recoveryErr, restoreErr)
}
}
if previousChanged {
var restoreErr error
if previousErr == nil {
restoreErr = replaceSymlink(single.PreviousLink, oldPrevious)
} else {
restoreErr = removeSymlink(single.PreviousLink)
}
recoveryErr = errors.Join(recoveryErr, restoreErr)
}
if handlerChanged && recoveryErr == nil {
if restoreErr := atomicWrite(single.CaddyHandler, oldHandler, 0o644); restoreErr != nil {
recoveryErr = errors.Join(recoveryErr, restoreErr)
} else if restoreErr = m.Operator.ValidateCaddy(ctx, single.CaddyConfig); restoreErr != nil {
recoveryErr = errors.Join(recoveryErr, restoreErr)
} else if restoreErr = m.Operator.ReloadCaddy(ctx); restoreErr != nil {
recoveryErr = errors.Join(recoveryErr, restoreErr)
}
}
if recoveryErr != nil && handlerChanged {
stopCandidate = false
err = &retainedCandidateError{cause: errors.Join(err, fmt.Errorf("singleton recovery incomplete; candidate remains routed for operator recovery: %w", recoveryErr))}
} }
}() }()
candidateHandler, err := renderHandler(single.CaddyHandlerTemplate, single.CandidateAddress)
if err != nil {
return err
}
if err = atomicWrite(single.CaddyHandler, candidateHandler, 0o644); err != nil {
return err
}
handlerChanged = true
if err = m.Operator.ValidateCaddy(ctx, single.CaddyConfig); err != nil {
return fmt.Errorf("candidate Caddy validation failed: %w", err)
}
if err = m.Operator.ReloadCaddy(ctx); err != nil {
return fmt.Errorf("candidate Caddy reload failed: %w", err)
}
if err = m.probePublic(ctx, cfg, checkMarkers); err != nil {
return fmt.Errorf("candidate public-origin smoke failed: %w", err)
}
if err = replaceSymlink(single.PreviousLink, oldCurrent); err != nil {
return err
}
previousChanged = true
if err = replaceSymlink(single.CurrentLink, release); err != nil {
return err
}
currentChanged = true
if err = m.Operator.Restart(ctx, single.Unit); err != nil { if err = m.Operator.Restart(ctx, single.Unit); err != nil {
return err return err
} }
if err = probeLocal(ctx, cfg, single.Address); err != nil { if err = m.probeHealthReadiness(ctx, cfg, single.Address); err != nil {
return fmt.Errorf("post-activation smoke failed: %w", err)
}
installedHandler, err := renderHandler(single.CaddyHandlerTemplate, single.Address)
if err != nil {
return err return err
} }
if err = atomicWrite(single.CaddyHandler, installedHandler, 0o644); err != nil { _ = replaceSymlink(single.PreviousLink, record.ActiveRelease)
return err next := state.Record{SchemaVersion: 1, Strategy: record.Strategy, ActiveSlot: "singleton", ActiveRelease: record.PreviousRelease, PreviousSlot: "singleton", PreviousRelease: record.ActiveRelease, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
} return state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next)
if err = m.Operator.ValidateCaddy(ctx, single.CaddyConfig); err != nil {
return fmt.Errorf("installed Caddy validation failed: %w", err)
}
if err = m.Operator.ReloadCaddy(ctx); err != nil {
return fmt.Errorf("installed Caddy reload failed: %w", err)
}
if err = m.probePublic(ctx, cfg, checkMarkers); err != nil {
return fmt.Errorf("public-origin smoke failed: %w", err)
}
if err = m.continuityWindow(ctx, cfg, single.CandidateAddress, checkMarkers); err != nil {
return fmt.Errorf("activation continuity failed: %w", err)
}
return nil
} }
func (m Manager) Status(ctx context.Context, cfg config.Config) (Status, error) { func (m Manager) Status(ctx context.Context, cfg config.Config) (Status, error) {
@@ -703,15 +356,6 @@ func (m Manager) Status(ctx context.Context, cfg config.Config) (Status, error)
units = []string{cfg.Deployment.BlueGreen.Blue.Unit, cfg.Deployment.BlueGreen.Green.Unit} units = []string{cfg.Deployment.BlueGreen.Blue.Unit, cfg.Deployment.BlueGreen.Green.Unit}
} else { } else {
units = []string{cfg.Deployment.Singleton.Unit} units = []string{cfg.Deployment.Singleton.Unit}
lease, exists, leaseErr := loadCandidateLease(cfg)
if leaseErr != nil {
return Status{}, leaseErr
}
if exists {
units = append(units, lease.Unit)
} else if result.StateInitialized && record.CandidateRelease != "" {
return Status{}, errors.New("candidate release has no operation-scoped lease; run tend reconcile --json")
}
} }
for _, unit := range units { for _, unit := range units {
active, err := m.Operator.IsActive(ctx, unit) active, err := m.Operator.IsActive(ctx, unit)
@@ -746,17 +390,6 @@ func (m Manager) Prune(cfg config.Config, keep int, apply bool) ([]string, error
} }
items := []candidate{} items := []candidate{}
protected := map[string]bool{record.ActiveRelease: true, record.PreviousRelease: true} protected := map[string]bool{record.ActiveRelease: true, record.PreviousRelease: true}
if cfg.Deployment.Strategy == "singleton_candidate" {
lease, exists, leaseErr := loadCandidateLease(cfg)
if leaseErr != nil {
return nil, leaseErr
}
if exists {
protected[lease.Release] = true
} else if record.CandidateRelease != "" {
protected[record.CandidateRelease] = true
}
}
for _, entry := range entries { for _, entry := range entries {
if !entry.IsDir() || entry.Type()&os.ModeSymlink != 0 || !strings.HasPrefix(entry.Name(), "sha256-") { if !entry.IsDir() || entry.Type()&os.ModeSymlink != 0 || !strings.HasPrefix(entry.Name(), "sha256-") {
continue continue
@@ -802,50 +435,6 @@ func (m Manager) probeHealthReadiness(ctx context.Context, cfg config.Config, ad
return m.probe(ctx, cfg, address, checks) return m.probe(ctx, cfg, address, checks)
} }
func (m Manager) probePublic(ctx context.Context, cfg config.Config, checkMarkers bool) error {
timeout := time.Duration(cfg.Deployment.CandidateTimeoutSecs) * time.Second
for _, check := range cfg.Deployment.PublicSmoke {
attempt, cancel := context.WithTimeout(ctx, timeout)
contains := check.Contains
if !checkMarkers {
contains = ""
}
err := m.Operator.ProbeURL(attempt, check.URL, contains)
cancel()
if err != nil {
return err
}
}
return nil
}
func (m Manager) continuityWindow(ctx context.Context, cfg config.Config, previousAddress string, checkMarkers bool) error {
steps := cfg.Deployment.ActivationWindowSecs * 4
if steps < 1 {
steps = 1
}
for step := 0; step < steps; step++ {
if err := m.probePublic(ctx, cfg, checkMarkers); err != nil {
return err
}
if previousAddress != "" {
if err := m.probeHealthReadiness(ctx, cfg, previousAddress); err != nil {
return fmt.Errorf("previous slot lost continuity: %w", err)
}
}
if step+1 < steps {
sleep := m.Sleep
if sleep == nil {
sleep = sleepContext
}
if err := sleep(ctx, 250*time.Millisecond); err != nil {
return err
}
}
}
return nil
}
func (m Manager) probe(ctx context.Context, cfg config.Config, address string, checks []config.Smoke) error { func (m Manager) probe(ctx context.Context, cfg config.Config, address string, checks []config.Smoke) error {
timeout := time.Duration(cfg.Deployment.CandidateTimeoutSecs) * time.Second timeout := time.Duration(cfg.Deployment.CandidateTimeoutSecs) * time.Second
for _, check := range checks { for _, check := range checks {
@@ -876,6 +465,13 @@ func slotConfig(bg config.BlueGreen, name string) config.Slot {
} }
return bg.Green return bg.Green
} }
func copyMap(source map[string]string) map[string]string {
target := make(map[string]string, len(source)+1)
for k, v := range source {
target[k] = v
}
return target
}
func resolveReleaseLink(root, link string) (string, error) { func resolveReleaseLink(root, link string) (string, error) {
info, err := os.Lstat(link) info, err := os.Lstat(link)
if err != nil { if err != nil {
@@ -892,7 +488,7 @@ func resolveReleaseLink(root, link string) (string, error) {
target = filepath.Join(filepath.Dir(link), target) target = filepath.Join(filepath.Dir(link), target)
} }
target = filepath.Clean(target) target = filepath.Clean(target)
probe := state.Record{SchemaVersion: state.SchemaVersion, Strategy: "singleton_candidate", ActiveSlot: "singleton", ActiveRelease: target, UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)} probe := state.Record{SchemaVersion: 1, Strategy: "singleton_candidate", ActiveSlot: "singleton", ActiveRelease: target, UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
if err := probe.Validate(root, "singleton_candidate"); err != nil { if err := probe.Validate(root, "singleton_candidate"); err != nil {
return "", err return "", err
} }
+22 -400
View File
@@ -5,7 +5,6 @@ package deploy
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -13,43 +12,21 @@ import (
"time" "time"
"gamertan.com/tend/internal/config" "gamertan.com/tend/internal/config"
"gamertan.com/tend/internal/eventlog"
"gamertan.com/tend/internal/state" "gamertan.com/tend/internal/state"
) )
const testCandidateUnit = "example-site-tend-candidate-dddddddddddd.service"
func storeTestCandidateLease(t *testing.T, cfg config.Config, release, startedAt string) {
t.Helper()
lease := state.CandidateLease{SchemaVersion: state.CandidateLeaseSchemaVersion, Service: cfg.Service.Name, OperationID: strings.Repeat("d", 32), Release: release, Unit: testCandidateUnit, Address: cfg.Deployment.Singleton.CandidateAddress, StartedAt: startedAt}
if err := state.StoreCandidateLease(state.CandidateLeasePath(cfg.Deployment.StateFile), cfg.Deployment.Root, lease); err != nil {
t.Fatal(err)
}
}
type fakeOperator struct { type fakeOperator struct {
failReload bool failReload bool
failReloadAt int
reloads int
failRestartUnit string failRestartUnit string
failRestartOnce bool
rejectMarkers bool rejectMarkers bool
active map[string]bool active map[string]bool
starts, stops, restarts []string starts, stops, restarts []string
probes []string probes []string
publicProbes []string
failPublic bool
failPublicAfter int
candidateEnvironment map[string]string
candidateFile string
} }
func (f *fakeOperator) Restart(_ context.Context, unit string) error { func (f *fakeOperator) Restart(_ context.Context, unit string) error {
f.restarts = append(f.restarts, unit) f.restarts = append(f.restarts, unit)
if unit == f.failRestartUnit { if unit == f.failRestartUnit {
if f.failRestartOnce {
f.failRestartUnit = ""
}
return errors.New("injected restart failure") return errors.New("injected restart failure")
} }
f.active[unit] = true f.active[unit] = true
@@ -63,92 +40,17 @@ func (f *fakeOperator) Stop(_ context.Context, unit string) error {
func (f *fakeOperator) IsActive(_ context.Context, unit string) (bool, error) { func (f *fakeOperator) IsActive(_ context.Context, unit string) (bool, error) {
return f.active[unit], nil return f.active[unit], nil
} }
func (f *fakeOperator) StartCandidate(_ context.Context, unit, binary, environmentFile string, env map[string]string) error { func (f *fakeOperator) StartCandidate(_ context.Context, unit, binary string, env map[string]string) error {
if !filepath.IsAbs(binary) || !filepath.IsAbs(environmentFile) || len(env) == 0 { if !filepath.IsAbs(binary) || len(env) == 0 {
return errors.New("bad candidate") return errors.New("bad candidate")
} }
f.starts = append(f.starts, unit) f.starts = append(f.starts, unit)
f.candidateFile = environmentFile
f.candidateEnvironment = make(map[string]string, len(env))
for key, value := range env {
f.candidateEnvironment[key] = value
}
f.active[unit] = true f.active[unit] = true
return nil return nil
} }
func (f *fakeOperator) ProbeURL(_ context.Context, value, contains string) error {
f.publicProbes = append(f.publicProbes, value)
if f.failPublic || (f.failPublicAfter > 0 && len(f.publicProbes) >= f.failPublicAfter) {
return errors.New("injected public smoke failure")
}
if f.rejectMarkers && contains != "" {
return errors.New("unexpected future-release smoke marker")
}
return nil
}
func TestPublicSmokeFailureRestoresBlueGreenHandlerAndSlot(t *testing.T) {
cfg, old, fresh := baseConfig(t, "blue_green")
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
template := filepath.Join(cfg.Deployment.Root, "handler.template")
original := []byte("reverse_proxy 127.0.0.1:8090\n")
_ = os.WriteFile(handler, original, 0o644)
_ = os.WriteFile(template, []byte("reverse_proxy {{UPSTREAM}}\n"), 0o644)
blue := filepath.Join(cfg.Deployment.Root, "slots", "blue")
green := filepath.Join(cfg.Deployment.Root, "slots", "green")
_ = replaceSymlink(blue, old)
_ = replaceSymlink(green, old)
cfg.Deployment.BlueGreen = &config.BlueGreen{CaddyConfig: filepath.Join(cfg.Deployment.Root, "Caddyfile"), CaddyHandler: handler, CaddyHandlerTemplate: template, BootstrapActive: "blue", Blue: config.Slot{Unit: "example-blue.service", Address: "127.0.0.1:8090", Link: blue}, Green: config.Slot{Unit: "example-green.service", Address: "127.0.0.1:8091", Link: green}}
operator := &fakeOperator{active: map[string]bool{}, failPublic: true}
if _, err := manager(operator, fresh).Deploy(context.Background(), cfg, Request{Activate: true}); err == nil {
t.Fatal("expected public smoke failure")
}
body, _ := os.ReadFile(handler)
if string(body) != string(original) {
t.Fatalf("handler not restored: %q", body)
}
target, err := resolveReleaseLink(cfg.Deployment.Root, green)
if err != nil || target != old {
t.Fatalf("green=%q err=%v", target, err)
}
record, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
if err != nil {
t.Fatal(err)
}
if record.ActiveRelease != old || record.LastAttemptOutcome != "failed" || record.CandidateRelease != "" || record.LastAttemptRelease != fresh {
t.Fatalf("failed attempt state=%+v", record)
}
}
func TestContinuityFailureRestoresBlueGreenRoute(t *testing.T) {
cfg, old, fresh := baseConfig(t, "blue_green")
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
template := filepath.Join(cfg.Deployment.Root, "handler.template")
original := []byte("reverse_proxy 127.0.0.1:8090\n")
_ = os.WriteFile(handler, original, 0o644)
_ = os.WriteFile(template, []byte("reverse_proxy {{UPSTREAM}}\n"), 0o644)
blue := filepath.Join(cfg.Deployment.Root, "slots", "blue")
green := filepath.Join(cfg.Deployment.Root, "slots", "green")
_ = replaceSymlink(blue, old)
_ = replaceSymlink(green, old)
cfg.Deployment.BlueGreen = &config.BlueGreen{CaddyConfig: filepath.Join(cfg.Deployment.Root, "Caddyfile"), CaddyHandler: handler, CaddyHandlerTemplate: template, BootstrapActive: "blue", Blue: config.Slot{Unit: "example-blue.service", Address: "127.0.0.1:8090", Link: blue}, Green: config.Slot{Unit: "example-green.service", Address: "127.0.0.1:8091", Link: green}}
operator := &fakeOperator{active: map[string]bool{}, failPublicAfter: 3}
if _, err := manager(operator, fresh).Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)}); err == nil || !strings.Contains(err.Error(), "continuity") {
t.Fatalf("expected continuity failure, got %v", err)
}
body, _ := os.ReadFile(handler)
if string(body) != string(original) {
t.Fatalf("handler not restored: %q", body)
}
target, err := resolveReleaseLink(cfg.Deployment.Root, green)
if err != nil || target != old {
t.Fatalf("green=%q err=%v", target, err)
}
}
func (f *fakeOperator) ValidateCaddy(context.Context, string) error { return nil } func (f *fakeOperator) ValidateCaddy(context.Context, string) error { return nil }
func (f *fakeOperator) ReloadCaddy(context.Context) error { func (f *fakeOperator) ReloadCaddy(context.Context) error {
f.reloads++ if f.failReload {
if f.failReload || (f.failReloadAt > 0 && f.reloads == f.failReloadAt) {
return errors.New("injected reload failure") return errors.New("injected reload failure")
} }
return nil return nil
@@ -167,7 +69,7 @@ func baseConfig(t *testing.T, strategy string) (config.Config, string, string) {
if err := os.MkdirAll(filepath.Join(root, "releases"), 0o755); err != nil { if err := os.MkdirAll(filepath.Join(root, "releases"), 0o755); err != nil {
t.Fatal(err) t.Fatal(err)
} }
old := filepath.Join(root, "releases", "sha256-"+strings.Repeat("c", 64)) old := filepath.Join(root, "releases", "legacy-old")
fresh := filepath.Join(root, "releases", "sha256-"+strings.Repeat("a", 64)) fresh := filepath.Join(root, "releases", "sha256-"+strings.Repeat("a", 64))
for _, dir := range []string{old, fresh} { for _, dir := range []string{old, fresh} {
if err := os.Mkdir(dir, 0o755); err != nil { if err := os.Mkdir(dir, 0o755); err != nil {
@@ -177,34 +79,11 @@ func baseConfig(t *testing.T, strategy string) (config.Config, string, string) {
t.Fatal(err) t.Fatal(err)
} }
} }
cfg := config.Config{SchemaVersion: 2, Service: config.Service{Name: "example-site", AllowedHost: "example.test", EnvironmentFile: "/etc/tend/environment/example-site.env"}, Build: config.Build{Package: "./cmd/site", Binary: "app", Branch: "main"}, Deployment: config.Deployment{Strategy: strategy, Root: root, LockFile: filepath.Join(root, "deploy.lock"), StateFile: filepath.Join(root, "state.json"), EventLog: filepath.Join(root, "deployment-events.jsonl"), HealthPath: "/healthz", ReadinessPath: "/readyz", CandidateTimeoutSecs: 2, ActivationWindowSecs: 1, Smoke: []config.Smoke{{Path: "/", Contains: "Example"}}, PublicSmoke: []config.PublicSmoke{{URL: "https://example.test/", Contains: "Example"}}}} cfg := config.Config{SchemaVersion: 1, Service: config.Service{Name: "example-site", AllowedHost: "example.test"}, Build: config.Build{Package: "./cmd/site", Binary: "app", Branch: "main"}, Deployment: config.Deployment{Strategy: strategy, Root: root, LockFile: filepath.Join(root, "deploy.lock"), StateFile: filepath.Join(root, "state.json"), HealthPath: "/healthz", ReadinessPath: "/readyz", CandidateTimeoutSecs: 2, Smoke: []config.Smoke{{Path: "/", Contains: "Example"}}}}
return cfg, old, fresh return cfg, old, fresh
} }
func manager(operator Operator, fresh string) Manager { func manager(operator Operator, fresh string) Manager {
return Manager{Operator: operator, Now: func() time.Time { return time.Unix(100, 0).UTC() }, Prepare: func(config.Config, string, string, string) (string, error) { return fresh, nil }, Inspect: func(config.Config, string, string, string) error { return nil }, ReadIdentity: func(string) (releaseIdentity, error) { return Manager{Operator: operator, Now: func() time.Time { return time.Unix(100, 0).UTC() }, Prepare: func(config.Config, string, string, string) (string, error) { return fresh, nil }, Inspect: func(config.Config, string, string, string) error { return nil }}
return releaseIdentity{Version: "v0.2.0-preview.1", Commit: strings.Repeat("b", 40)}, nil
}, OperationID: func() (string, error) { return strings.Repeat("d", 32), nil }, Sleep: func(context.Context, time.Duration) error { return nil }}
}
func singletonSettings(t *testing.T, cfg config.Config, currentRelease string) (*config.Singleton, string) {
t.Helper()
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
template := filepath.Join(cfg.Deployment.Root, "handler.template")
if err := os.WriteFile(handler, []byte("reverse_proxy 127.0.0.1:8092\n"), 0o640); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(template, []byte("reverse_proxy {{UPSTREAM}}\n"), 0o644); err != nil {
t.Fatal(err)
}
current := filepath.Join(cfg.Deployment.Root, "current")
if err := replaceSymlink(current, currentRelease); err != nil {
t.Fatal(err)
}
return &config.Singleton{
Unit: "example-site.service", Address: "127.0.0.1:8092", CandidateAddress: "127.0.0.1:18092", ListenEnv: "EXAMPLE_LISTEN",
CurrentLink: current, PreviousLink: filepath.Join(cfg.Deployment.Root, "previous"), CaddyConfig: filepath.Join(cfg.Deployment.Root, "Caddyfile"),
CaddyHandler: handler, CaddyHandlerTemplate: template,
}, handler
} }
func TestBlueGreenActivationAndRollback(t *testing.T) { func TestBlueGreenActivationAndRollback(t *testing.T) {
@@ -231,28 +110,13 @@ func TestBlueGreenActivationAndRollback(t *testing.T) {
} }
operator := &fakeOperator{active: map[string]bool{"example-blue.service": true, "example-green.service": true}} operator := &fakeOperator{active: map[string]bool{"example-blue.service": true, "example-green.service": true}}
m := manager(operator, fresh) m := manager(operator, fresh)
var events []eventlog.Event report, err := m.Deploy(context.Background(), cfg, Request{Activate: true})
m.AppendEvent = func(_ string, event eventlog.Event) error {
events = append(events, event)
return nil
}
report, err := m.Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)})
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if report.ActiveRelease != fresh || report.PreviousRelease != old { if report.ActiveRelease != fresh || report.PreviousRelease != old {
t.Fatalf("report=%+v", report) t.Fatalf("report=%+v", report)
} }
deployed, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
if err != nil {
t.Fatal(err)
}
if deployed.DesiredRelease != fresh || deployed.CandidateRelease != "" || deployed.LastAttemptRelease != fresh || deployed.LastAttemptOutcome != "succeeded" {
t.Fatalf("deployment identity state=%+v", deployed)
}
if len(events) != 2 || events[0].Phase != "candidate" || events[0].Outcome != "running" || events[1].Phase != "activation" || events[1].Outcome != "succeeded" || events[0].OperationID != events[1].OperationID {
t.Fatalf("events=%+v", events)
}
if info, err := os.Stat(handler); err != nil || info.Mode().Perm() != 0o640 { if info, err := os.Stat(handler); err != nil || info.Mode().Perm() != 0o640 {
t.Fatalf("handler mode=%v err=%v", info.Mode().Perm(), err) t.Fatalf("handler mode=%v err=%v", info.Mode().Perm(), err)
} }
@@ -268,9 +132,6 @@ func TestBlueGreenActivationAndRollback(t *testing.T) {
if record.ActiveRelease != old || record.PreviousRelease != fresh { if record.ActiveRelease != old || record.PreviousRelease != fresh {
t.Fatalf("rollback=%+v", record) t.Fatalf("rollback=%+v", record)
} }
if len(events) != 4 || events[2].Phase != "rollback" || events[2].Outcome != "running" || events[3].Phase != "rollback" || events[3].Outcome != "succeeded" || events[2].OperationID != events[3].OperationID || events[2].ArtifactDigest != strings.Repeat("c", 64) {
t.Fatalf("rollback events=%+v", events)
}
if len(operator.probes) != 4 { if len(operator.probes) != 4 {
t.Fatalf("rollback probes=%#v", operator.probes) t.Fatalf("rollback probes=%#v", operator.probes)
} }
@@ -281,76 +142,6 @@ func TestBlueGreenActivationAndRollback(t *testing.T) {
} }
} }
func TestDeploymentEvidenceCanNeverBlockActivationOrRollback(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate")
cfg.Deployment.Singleton, _ = singletonSettings(t, cfg, old)
m := manager(&fakeOperator{active: map[string]bool{}}, fresh)
appendCalls := 0
m.AppendEvent = func(string, eventlog.Event) error {
appendCalls++
return errors.New("injected event failure")
}
report, err := m.Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)})
if err != nil {
t.Fatal(err)
}
if report.EventWarnings != 2 || report.ActiveRelease != fresh {
t.Fatalf("report=%+v", report)
}
if _, err := m.Rollback(context.Background(), cfg); err != nil {
t.Fatal(err)
}
record, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
if err != nil || record.ActiveRelease != old || appendCalls != 4 {
t.Fatalf("record=%+v appends=%d err=%v", record, appendCalls, err)
}
}
func TestDeploymentEvidenceIdentityIsBestEffort(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate")
cfg.Deployment.Singleton, _ = singletonSettings(t, cfg, old)
m := manager(&fakeOperator{active: map[string]bool{}}, fresh)
m.ReadIdentity = func(string) (releaseIdentity, error) {
return releaseIdentity{}, errors.New("injected identity failure")
}
report, err := m.Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)})
if err != nil || report.EventWarnings != 1 || report.ActiveRelease != fresh {
t.Fatalf("report=%+v err=%v", report, err)
}
}
func TestSingletonOperationIdentityIsRequiredBeforeCandidateStart(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate")
cfg.Deployment.Singleton, _ = singletonSettings(t, cfg, old)
operator := &fakeOperator{active: map[string]bool{}}
m := manager(operator, fresh)
m.OperationID = func() (string, error) { return "", errors.New("injected entropy failure") }
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)}); err == nil || !strings.Contains(err.Error(), "operation identity") {
t.Fatalf("expected operation identity refusal, got %v", err)
}
if len(operator.starts) != 0 {
t.Fatalf("candidate started without an operation identity: %#v", operator.starts)
}
}
func TestSingletonUnresolvedLeaseBlocksReplacementBeforeCandidateStart(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate")
cfg.Deployment.Singleton, _ = singletonSettings(t, cfg, old)
at := time.Unix(100, 0).UTC().Format(time.RFC3339)
record := state.Record{SchemaVersion: state.SchemaVersion, Strategy: cfg.Deployment.Strategy, DesiredRelease: fresh, CandidateRelease: fresh, ActiveSlot: "singleton", ActiveRelease: old, LastAttemptRelease: fresh, LastAttemptOutcome: "failed", LastAttemptAt: at, UpdatedAt: at}
if err := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, record); err != nil {
t.Fatal(err)
}
storeTestCandidateLease(t, cfg, fresh, at)
operator := &fakeOperator{active: map[string]bool{testCandidateUnit: true}}
if _, err := manager(operator, fresh).Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)}); err == nil || !strings.Contains(err.Error(), "run tend reconcile") {
t.Fatalf("expected unresolved lease refusal, got %v", err)
}
if len(operator.starts) != 0 || !operator.active[testCandidateUnit] {
t.Fatalf("existing candidate was disturbed: starts=%#v active=%#v", operator.starts, operator.active)
}
}
func TestBlueGreenCaddyFailureRestoresHandlerAndSlot(t *testing.T) { func TestBlueGreenCaddyFailureRestoresHandlerAndSlot(t *testing.T) {
cfg, old, fresh := baseConfig(t, "blue_green") cfg, old, fresh := baseConfig(t, "blue_green")
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy") handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
@@ -376,47 +167,37 @@ func TestBlueGreenCaddyFailureRestoresHandlerAndSlot(t *testing.T) {
if err != nil || target != old { if err != nil || target != old {
t.Fatalf("green=%q err=%v", target, err) t.Fatalf("green=%q err=%v", target, err)
} }
failed, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy) if _, err := os.Stat(cfg.Deployment.StateFile); !os.IsNotExist(err) {
if err != nil { t.Fatal("failed activation wrote state")
t.Fatal(err)
}
if failed.ActiveRelease != old || failed.LastAttemptOutcome != "failed" || failed.CandidateRelease != "" {
t.Fatalf("failed state=%+v", failed)
} }
} }
func TestSingletonRestartFailureRestoresPointers(t *testing.T) { func TestSingletonRestartFailureRestoresPointers(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate") cfg, old, fresh := baseConfig(t, "singleton_candidate")
settings, handler := singletonSettings(t, cfg, old) previous := filepath.Join(cfg.Deployment.Root, "previous")
cfg.Deployment.Singleton = settings current := filepath.Join(cfg.Deployment.Root, "current")
operator := &fakeOperator{active: map[string]bool{}, failRestartUnit: "example-site.service", failRestartOnce: true} _ = replaceSymlink(current, old)
cfg.Deployment.Singleton = &config.Singleton{Unit: "example-site.service", Address: "127.0.0.1:8092", CandidateAddress: "127.0.0.1:18092", ListenEnv: "EXAMPLE_LISTEN", CurrentLink: current, PreviousLink: previous}
operator := &fakeOperator{active: map[string]bool{}, failRestartUnit: "example-site.service"}
m := manager(operator, fresh) m := manager(operator, fresh)
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err == nil { if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err == nil {
t.Fatal("expected failure") t.Fatal("expected failure")
} }
target, err := resolveReleaseLink(cfg.Deployment.Root, cfg.Deployment.Singleton.CurrentLink) target, err := resolveReleaseLink(cfg.Deployment.Root, current)
if err != nil || target != old { if err != nil || target != old {
t.Fatalf("current=%q err=%v", target, err) t.Fatalf("current=%q err=%v", target, err)
} }
if _, err := os.Lstat(cfg.Deployment.Singleton.PreviousLink); !os.IsNotExist(err) { if _, err := os.Lstat(previous); !os.IsNotExist(err) {
t.Fatal("previous pointer was not restored") t.Fatal("previous pointer was not restored")
} }
body, err := os.ReadFile(handler)
if err != nil || string(body) != "reverse_proxy 127.0.0.1:8092\n" {
t.Fatalf("handler=%q err=%v", body, err)
}
if info, err := os.Stat(handler); err != nil || info.Mode().Perm() != 0o640 {
t.Fatalf("handler mode=%v err=%v", info.Mode().Perm(), err)
}
if operator.active[testCandidateUnit] {
t.Fatal("candidate was not stopped after successful restoration")
}
} }
func TestStateRecordsSuccessfulActivation(t *testing.T) { func TestStatePersistsOnlyAfterSuccessfulActivation(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate") cfg, old, fresh := baseConfig(t, "singleton_candidate")
settings, handler := singletonSettings(t, cfg, old) current := filepath.Join(cfg.Deployment.Root, "current")
cfg.Deployment.Singleton = settings previous := filepath.Join(cfg.Deployment.Root, "previous")
_ = replaceSymlink(current, old)
cfg.Deployment.Singleton = &config.Singleton{Unit: "example-site.service", Address: "127.0.0.1:8092", CandidateAddress: "127.0.0.1:18092", ListenEnv: "EXAMPLE_LISTEN", CurrentLink: current, PreviousLink: previous}
operator := &fakeOperator{active: map[string]bool{}} operator := &fakeOperator{active: map[string]bool{}}
m := manager(operator, fresh) m := manager(operator, fresh)
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err != nil { if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err != nil {
@@ -426,166 +207,7 @@ func TestStateRecordsSuccessfulActivation(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if record.ActiveRelease != fresh || record.PreviousRelease != old || record.CandidateRelease != "" { if record.ActiveRelease != fresh || record.PreviousRelease != old {
t.Fatalf("state=%+v", record) t.Fatalf("state=%+v", record)
} }
if _, err := os.Lstat(state.CandidateLeasePath(cfg.Deployment.StateFile)); !os.IsNotExist(err) {
t.Fatalf("candidate lease was not removed: %v", err)
}
if len(operator.starts) != 1 || operator.starts[0] != testCandidateUnit {
t.Fatalf("candidate starts=%#v", operator.starts)
}
if operator.candidateFile != cfg.Service.EnvironmentFile || len(operator.candidateEnvironment) != 1 || operator.candidateEnvironment["EXAMPLE_LISTEN"] != "127.0.0.1:18092" {
t.Fatalf("candidate file=%q environment=%#v", operator.candidateFile, operator.candidateEnvironment)
}
body, err := os.ReadFile(handler)
if err != nil || string(body) != "reverse_proxy 127.0.0.1:8092\n" {
t.Fatalf("handler=%q err=%v", body, err)
}
if operator.reloads != 2 || operator.active[testCandidateUnit] {
t.Fatalf("reloads=%d active=%#v", operator.reloads, operator.active)
}
reconciliation, err := m.Reconcile(context.Background(), cfg)
if err != nil || reconciliation.Mutation != "none" || reconciliation.Disposition != "settled" || reconciliation.Observed.CandidateLease || !reconciliation.Consistent {
t.Fatalf("reconciliation=%+v err=%v", reconciliation, err)
}
}
func TestSingletonContinuityFailureRestoresHandlerPointersAndService(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate")
settings, handler := singletonSettings(t, cfg, old)
cfg.Deployment.Singleton = settings
operator := &fakeOperator{active: map[string]bool{"example-site.service": true}, failPublicAfter: 4}
m := manager(operator, fresh)
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err == nil || !strings.Contains(err.Error(), "continuity") {
t.Fatalf("expected continuity failure, got %v", err)
}
current, err := resolveReleaseLink(cfg.Deployment.Root, cfg.Deployment.Singleton.CurrentLink)
if err != nil || current != old {
t.Fatalf("current=%q err=%v", current, err)
}
body, err := os.ReadFile(handler)
if err != nil || string(body) != "reverse_proxy 127.0.0.1:8092\n" {
t.Fatalf("handler=%q err=%v", body, err)
}
if operator.active[testCandidateUnit] {
t.Fatal("candidate was not stopped after continuity restoration")
}
}
func TestSingletonCaddyReloadFailuresRestorePriorRoute(t *testing.T) {
for _, reload := range []int{1, 2} {
t.Run(fmt.Sprintf("reload-%d", reload), func(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate")
settings, handler := singletonSettings(t, cfg, old)
cfg.Deployment.Singleton = settings
operator := &fakeOperator{active: map[string]bool{"example-site.service": true}, failReloadAt: reload}
if _, err := manager(operator, fresh).Deploy(context.Background(), cfg, Request{Activate: true}); err == nil || !strings.Contains(err.Error(), "Caddy reload failed") {
t.Fatalf("expected Caddy reload failure, got %v", err)
}
current, err := resolveReleaseLink(cfg.Deployment.Root, settings.CurrentLink)
if err != nil || current != old {
t.Fatalf("current=%q err=%v", current, err)
}
body, err := os.ReadFile(handler)
if err != nil || string(body) != "reverse_proxy 127.0.0.1:8092\n" {
t.Fatalf("handler=%q err=%v", body, err)
}
if operator.active[testCandidateUnit] {
t.Fatal("candidate was not stopped after route restoration")
}
})
}
}
func TestSingletonIncompleteRecoveryKeepsProvenCandidateRunning(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate")
settings, _ := singletonSettings(t, cfg, old)
cfg.Deployment.Singleton = settings
operator := &fakeOperator{active: map[string]bool{"example-site.service": true}, failReload: true}
_, err := manager(operator, fresh).Deploy(context.Background(), cfg, Request{Activate: true})
if err == nil || !strings.Contains(err.Error(), "candidate remains routed for operator recovery") {
t.Fatalf("expected explicit incomplete recovery, got %v", err)
}
if !operator.active[testCandidateUnit] {
t.Fatal("proven candidate was stopped despite incomplete route restoration")
}
record, loadErr := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
lease, leaseErr := state.LoadCandidateLease(state.CandidateLeasePath(cfg.Deployment.StateFile), cfg.Deployment.Root, cfg.Service.Name)
if loadErr != nil || leaseErr != nil || record.CandidateRelease != fresh || lease.OperationID != strings.Repeat("d", 32) || lease.Unit != testCandidateUnit {
t.Fatalf("retained state=%+v err=%v", record, loadErr)
}
}
func TestReconcileReportsRetainedCandidateWithoutMutation(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate")
settings, handler := singletonSettings(t, cfg, old)
cfg.Deployment.Singleton = settings
at := time.Unix(100, 0).UTC().Format(time.RFC3339)
record := state.Record{SchemaVersion: state.SchemaVersion, Strategy: cfg.Deployment.Strategy, DesiredRelease: fresh, CandidateRelease: fresh, ActiveSlot: "singleton", ActiveRelease: old, LastAttemptRelease: fresh, LastAttemptOutcome: "failed", LastAttemptAt: at, UpdatedAt: at}
if err := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, record); err != nil {
t.Fatal(err)
}
storeTestCandidateLease(t, cfg, fresh, at)
candidateHandler, err := renderHandler(settings.CaddyHandlerTemplate, settings.CandidateAddress)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(handler, candidateHandler, 0o640); err != nil {
t.Fatal(err)
}
operator := &fakeOperator{active: map[string]bool{settings.Unit: true, testCandidateUnit: true}}
reconciliation, err := manager(operator, fresh).Reconcile(context.Background(), cfg)
if err != nil || reconciliation.Mutation != "none" || !reconciliation.Observed.CandidateLease || reconciliation.Observed.CandidateUnitActive == nil || !*reconciliation.Observed.CandidateUnitActive || reconciliation.Observed.HandlerFileTarget != "candidate" || reconciliation.Disposition != "retained_candidate_handler_file" {
t.Fatalf("reconciliation=%+v err=%v", reconciliation, err)
}
if len(operator.stops) != 0 || len(operator.restarts) != 0 || operator.reloads != 0 {
t.Fatalf("reconcile mutated services: stops=%#v restarts=%#v reloads=%d", operator.stops, operator.restarts, operator.reloads)
}
}
func TestSingletonRollbackRetainsOperationLeaseWhenRecoveryIsIncomplete(t *testing.T) {
cfg, old, fresh := baseConfig(t, "singleton_candidate")
cfg.Deployment.Singleton, _ = singletonSettings(t, cfg, old)
operator := &fakeOperator{active: map[string]bool{cfg.Deployment.Singleton.Unit: true}}
m := manager(operator, fresh)
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)}); err != nil {
t.Fatal(err)
}
operator.failReload = true
if _, err := m.Rollback(context.Background(), cfg); err == nil || !strings.Contains(err.Error(), "candidate remains routed for operator recovery") {
t.Fatalf("expected retained rollback candidate, got %v", err)
}
record, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
if err != nil {
t.Fatal(err)
}
lease, leaseErr := state.LoadCandidateLease(state.CandidateLeasePath(cfg.Deployment.StateFile), cfg.Deployment.Root, cfg.Service.Name)
if record.ActiveRelease != fresh || record.CandidateRelease != old || leaseErr != nil || lease.OperationID != strings.Repeat("d", 32) || lease.Unit != testCandidateUnit || record.LastAttemptOutcome != "failed" {
t.Fatalf("rollback state=%+v", record)
}
if !operator.active[testCandidateUnit] {
t.Fatal("rollback candidate was stopped despite incomplete recovery")
}
}
func TestPruneProtectsOperationScopedCandidateRelease(t *testing.T) {
cfg, active, candidate := baseConfig(t, "singleton_candidate")
cfg.Deployment.Singleton, _ = singletonSettings(t, cfg, active)
at := time.Unix(100, 0).UTC().Format(time.RFC3339)
record := state.Record{SchemaVersion: state.SchemaVersion, Strategy: cfg.Deployment.Strategy, DesiredRelease: candidate, CandidateRelease: candidate, ActiveSlot: "singleton", ActiveRelease: active, LastAttemptRelease: candidate, LastAttemptOutcome: "failed", LastAttemptAt: at, UpdatedAt: at}
if err := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, record); err != nil {
t.Fatal(err)
}
storeTestCandidateLease(t, cfg, candidate, at)
removed, err := manager(&fakeOperator{active: map[string]bool{}}, candidate).Prune(cfg, 2, true)
if err != nil {
t.Fatal(err)
}
if len(removed) != 0 {
t.Fatalf("candidate release was selected for pruning: %#v", removed)
}
if info, err := os.Stat(candidate); err != nil || !info.IsDir() {
t.Fatalf("candidate release was not preserved: %v", err)
}
} }
-30
View File
@@ -1,30 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
//go:build linux
package deploy
import (
"path/filepath"
"testing"
)
func TestHostWideLockSerializesIndependentServices(t *testing.T) {
path := filepath.Join(t.TempDir(), "tend-deploy.lock")
first, err := acquireLock(path)
if err != nil {
t.Fatal(err)
}
defer first.Close()
if second, err := acquireLock(path); err == nil {
_ = second.Close()
t.Fatal("second service acquired the shared activation lock")
}
if err = first.Close(); err != nil {
t.Fatal(err)
}
third, err := acquireLock(path)
if err != nil {
t.Fatal(err)
}
_ = third.Close()
}
+3 -17
View File
@@ -20,11 +20,10 @@ type Operator interface {
Restart(context.Context, string) error Restart(context.Context, string) error
Stop(context.Context, string) error Stop(context.Context, string) error
IsActive(context.Context, string) (bool, error) IsActive(context.Context, string) (bool, error)
StartCandidate(context.Context, string, string, string, map[string]string) error StartCandidate(context.Context, string, string, map[string]string) error
ValidateCaddy(context.Context, string) error ValidateCaddy(context.Context, string) error
ReloadCaddy(context.Context) error ReloadCaddy(context.Context) error
Probe(context.Context, string, string, string, string) error Probe(context.Context, string, string, string, string) error
ProbeURL(context.Context, string, string) error
} }
type SystemOperator struct { type SystemOperator struct {
@@ -50,7 +49,7 @@ func (o SystemOperator) IsActive(ctx context.Context, unit string) (bool, error)
} }
return strings.TrimSpace(string(out)) == "active", nil return strings.TrimSpace(string(out)) == "active", nil
} }
func (o SystemOperator) StartCandidate(ctx context.Context, unit, binary, environmentFile string, env map[string]string) error { func (o SystemOperator) StartCandidate(ctx context.Context, unit, binary string, env map[string]string) error {
args := []string{ args := []string{
"--unit", unit, "--collect", "--unit", unit, "--collect",
"--property=DynamicUser=yes", "--property=NoNewPrivileges=yes", "--property=DynamicUser=yes", "--property=NoNewPrivileges=yes",
@@ -64,7 +63,6 @@ func (o SystemOperator) StartCandidate(ctx context.Context, unit, binary, enviro
"--property=RestrictSUIDSGID=yes", "--property=LockPersonality=yes", "--property=RestrictSUIDSGID=yes", "--property=LockPersonality=yes",
"--property=MemoryDenyWriteExecute=yes", "--property=CapabilityBoundingSet=", "--property=MemoryDenyWriteExecute=yes", "--property=CapabilityBoundingSet=",
"--property=AmbientCapabilities=", "--property=AmbientCapabilities=",
"--property=EnvironmentFile=" + environmentFile,
} }
keys := make([]string, 0, len(env)) keys := make([]string, 0, len(env))
for key := range env { for key := range env {
@@ -88,23 +86,11 @@ func (o SystemOperator) ReloadCaddy(ctx context.Context) error {
} }
func (o SystemOperator) Probe(ctx context.Context, address, host, path, contains string) error { func (o SystemOperator) Probe(ctx context.Context, address, host, path, contains string) error {
u := url.URL{Scheme: "http", Host: address, Path: path} u := url.URL{Scheme: "http", Host: address, Path: path}
return o.probeRequest(ctx, u.String(), host, contains) req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
}
func (o SystemOperator) ProbeURL(ctx context.Context, value, contains string) error {
u, err := url.Parse(value)
if err != nil || u.Scheme != "https" || u.Host == "" || u.User != nil || u.Fragment != "" {
return errors.New("public probe URL is invalid")
}
return o.probeRequest(ctx, u.String(), "", contains)
}
func (o SystemOperator) probeRequest(ctx context.Context, value, host, contains string) error {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, value, nil)
if err != nil { if err != nil {
return err return err
} }
if host != "" {
req.Host = host req.Host = host
}
client := &http.Client{Timeout: o.Timeout, CheckRedirect: func(*http.Request, []*http.Request) error { return errors.New("redirect refused") }} client := &http.Client{Timeout: o.Timeout, CheckRedirect: func(*http.Request, []*http.Request) error { return errors.New("redirect refused") }}
response, err := client.Do(req) response, err := client.Do(req)
if err != nil { if err != nil {
+2 -6
View File
@@ -5,7 +5,6 @@ package deploy
import ( import (
"context" "context"
"reflect" "reflect"
"strings"
"testing" "testing"
) )
@@ -24,13 +23,13 @@ func TestStartCandidateUsesArgumentVectorAndHardenedUnit(t *testing.T) {
runner := &recordingRunner{} runner := &recordingRunner{}
operator := SystemOperator{Runner: runner} operator := SystemOperator{Runner: runner}
env := map[string]string{"Z_ENV": "safe value", "A_ENV": "first"} env := map[string]string{"Z_ENV": "safe value", "A_ENV": "first"}
if err := operator.StartCandidate(context.Background(), "example-tend-candidate.service", "/opt/example/releases/sha256-a/app", "/etc/tend/environment/example.env", env); err != nil { if err := operator.StartCandidate(context.Background(), "example-tend-candidate.service", "/opt/example/releases/sha256-a/app", env); err != nil {
t.Fatal(err) t.Fatal(err)
} }
if runner.name != "systemd-run" { if runner.name != "systemd-run" {
t.Fatalf("command=%q", runner.name) t.Fatalf("command=%q", runner.name)
} }
required := []string{"--property=DynamicUser=yes", "--property=NoNewPrivileges=yes", "--property=ProtectSystem=strict", "--property=MemoryDenyWriteExecute=yes", "--property=CapabilityBoundingSet=", "--property=EnvironmentFile=/etc/tend/environment/example.env", "--setenv", "A_ENV=first", "--setenv", "Z_ENV=safe value", "--", "/opt/example/releases/sha256-a/app"} required := []string{"--property=DynamicUser=yes", "--property=NoNewPrivileges=yes", "--property=ProtectSystem=strict", "--property=MemoryDenyWriteExecute=yes", "--property=CapabilityBoundingSet=", "--setenv", "A_ENV=first", "--setenv", "Z_ENV=safe value", "--", "/opt/example/releases/sha256-a/app"}
cursor := 0 cursor := 0
for _, arg := range runner.args { for _, arg := range runner.args {
if cursor < len(required) && arg == required[cursor] { if cursor < len(required) && arg == required[cursor] {
@@ -43,7 +42,4 @@ func TestStartCandidateUsesArgumentVectorAndHardenedUnit(t *testing.T) {
if reflect.DeepEqual(runner.args, []string{"sh", "-c"}) { if reflect.DeepEqual(runner.args, []string{"sh", "-c"}) {
t.Fatal("candidate command used a shell") t.Fatal("candidate command used a shell")
} }
if strings.Contains(strings.Join(runner.args, "\n"), "SUPER_SECRET") {
t.Fatal("candidate arguments exposed a secret value")
}
} }
-270
View File
@@ -1,270 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package deploy
import (
"bytes"
"context"
"errors"
"os"
"gamertan.com/tend/internal/config"
"gamertan.com/tend/internal/state"
)
type Finding struct {
Code string `json:"code"`
Severity string `json:"severity"`
Message string `json:"message"`
}
type ObservedReleaseIdentity struct {
Version string `json:"version"`
Commit string `json:"commit"`
}
type ObservedState struct {
ActiveRelease string `json:"active_release,omitempty"`
PreviousRelease string `json:"previous_release,omitempty"`
ActiveIdentity *ObservedReleaseIdentity `json:"active_identity,omitempty"`
Units map[string]bool `json:"units"`
CandidateLease bool `json:"candidate_lease"`
CandidateUnit string `json:"candidate_unit,omitempty"`
CandidateUnitActive *bool `json:"candidate_unit_active,omitempty"`
LegacyCandidateUnit string `json:"legacy_candidate_unit,omitempty"`
LegacyCandidateActive *bool `json:"legacy_candidate_unit_active,omitempty"`
RouteHandlerMatches bool `json:"route_handler_matches"`
HandlerFileTarget string `json:"handler_file_target,omitempty"`
}
type Reconciliation struct {
Service string `json:"service"`
Strategy string `json:"strategy"`
Mutation string `json:"mutation"`
Consistent bool `json:"consistent"`
StateInitialized bool `json:"state_initialized"`
State *state.Record `json:"state,omitempty"`
Observed ObservedState `json:"observed"`
Disposition string `json:"disposition"`
Findings []Finding `json:"findings"`
}
// Reconcile observes configured units, release pointers, installed release
// identity, and the imported Caddy handler. It never acquires the deployment
// lock or mutates service state; proposed repairs remain an operator decision.
func (m Manager) Reconcile(ctx context.Context, cfg config.Config) (Reconciliation, error) {
if err := cfg.Validate(); err != nil {
return Reconciliation{}, err
}
if m.Operator == nil || m.ReadIdentity == nil {
return Reconciliation{}, errors.New("reconciliation dependencies are unavailable")
}
report := Reconciliation{
Service: cfg.Service.Name, Strategy: cfg.Deployment.Strategy, Mutation: "none",
Observed: ObservedState{Units: map[string]bool{}}, Findings: []Finding{},
}
add := func(code, severity, message string) {
report.Findings = append(report.Findings, Finding{Code: code, Severity: severity, Message: message})
}
record, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
if err == nil {
report.State = &record
report.StateInitialized = true
} else if os.IsNotExist(err) {
add("state_uninitialized", "warning", "Tend has no validated state record for this service.")
} else {
add("state_invalid", "error", "The Tend state record could not be validated.")
}
activeSlot := ""
activeUnit := ""
activeAddress := ""
handler := ""
template := ""
var candidateLease *state.CandidateLease
switch cfg.Deployment.Strategy {
case "singleton_candidate":
single := *cfg.Deployment.Singleton
activeSlot = "singleton"
activeUnit = single.Unit
activeAddress = single.Address
handler, template = single.CaddyHandler, single.CaddyHandlerTemplate
report.Observed.ActiveRelease = observeReleaseLink(cfg, single.CurrentLink, true, add)
report.Observed.PreviousRelease = observeReleaseLink(cfg, single.PreviousLink, false, add)
legacyCandidateUnit := cfg.Service.Name + "-tend-candidate.service"
candidateUnit := legacyCandidateUnit
lease, leaseErr := state.LoadCandidateLease(state.CandidateLeasePath(cfg.Deployment.StateFile), cfg.Deployment.Root, cfg.Service.Name)
if leaseErr == nil {
candidateLease = &lease
report.Observed.CandidateLease = true
candidateUnit = lease.Unit
} else if !os.IsNotExist(leaseErr) {
report.Observed.CandidateLease = true
candidateUnit = ""
add("candidate_lease_invalid", "error", "The operation-scoped candidate lease could not be validated.")
} else if report.State != nil && report.State.CandidateRelease != "" {
report.Observed.CandidateLease = true
candidateUnit = ""
add("legacy_candidate_lease", "error", "The state records a candidate release without an operation-scoped lease and requires manual review.")
}
if candidateUnit != "" {
report.Observed.CandidateUnit = candidateUnit
candidateActive, candidateErr := m.Operator.IsActive(ctx, candidateUnit)
if candidateErr != nil {
add("candidate_unit_unobservable", "error", "The transient candidate unit state could not be observed.")
} else {
report.Observed.CandidateUnitActive = &candidateActive
report.Observed.Units[candidateUnit] = candidateActive
}
}
if candidateUnit != legacyCandidateUnit {
report.Observed.LegacyCandidateUnit = legacyCandidateUnit
legacyActive, legacyErr := m.Operator.IsActive(ctx, legacyCandidateUnit)
if legacyErr != nil {
add("legacy_candidate_unit_unobservable", "error", "The legacy fixed candidate unit state could not be observed.")
} else {
report.Observed.LegacyCandidateActive = &legacyActive
report.Observed.Units[legacyCandidateUnit] = legacyActive
if legacyActive {
add("legacy_candidate_unit_active", "error", "A legacy fixed-name candidate remains active beside an operation-scoped lease.")
}
}
}
case "blue_green":
blueGreen := *cfg.Deployment.BlueGreen
handler, template = blueGreen.CaddyHandler, blueGreen.CaddyHandlerTemplate
activeSlot = blueGreen.BootstrapActive
if report.State != nil {
activeSlot = report.State.ActiveSlot
}
active := slotConfig(blueGreen, activeSlot)
previousName := "blue"
if activeSlot == "blue" {
previousName = "green"
}
previous := slotConfig(blueGreen, previousName)
activeUnit, activeAddress = active.Unit, active.Address
report.Observed.ActiveRelease = observeReleaseLink(cfg, active.Link, true, add)
report.Observed.PreviousRelease = observeReleaseLink(cfg, previous.Link, false, add)
for _, slot := range []config.Slot{blueGreen.Blue, blueGreen.Green} {
observeUnit(ctx, m.Operator, slot.Unit, report.Observed.Units, add)
}
}
if _, exists := report.Observed.Units[activeUnit]; !exists {
observeUnit(ctx, m.Operator, activeUnit, report.Observed.Units, add)
}
if active, observed := report.Observed.Units[activeUnit]; activeUnit != "" && observed && !active {
add("active_unit_inactive", "error", "The configured active service unit is not active.")
}
if report.Observed.ActiveRelease != "" {
identity, identityErr := m.ReadIdentity(report.Observed.ActiveRelease)
if identityErr != nil {
add("active_identity_unreadable", "error", "The observed active release identity could not be validated.")
} else {
report.Observed.ActiveIdentity = &ObservedReleaseIdentity{Version: identity.Version, Commit: identity.Commit}
}
}
expectedHandler, renderErr := renderHandler(template, activeAddress)
actualHandler, readErr := os.ReadFile(handler)
if renderErr != nil || readErr != nil {
add("route_handler_unreadable", "error", "The configured Caddy handler or its template could not be validated.")
} else {
report.Observed.RouteHandlerMatches = bytes.Equal(expectedHandler, actualHandler)
if report.Observed.RouteHandlerMatches {
report.Observed.HandlerFileTarget = "installed"
} else if cfg.Deployment.Strategy == "singleton_candidate" {
candidateAddress := cfg.Deployment.Singleton.CandidateAddress
if candidateLease != nil {
candidateAddress = candidateLease.Address
}
candidateHandler, candidateErr := renderHandler(template, candidateAddress)
if candidateErr == nil && bytes.Equal(candidateHandler, actualHandler) {
report.Observed.HandlerFileTarget = "candidate"
} else {
report.Observed.HandlerFileTarget = "other"
}
}
if !report.Observed.RouteHandlerMatches {
add("route_handler_drift", "error", "The installed Caddy handler does not match the configured active upstream.")
}
}
if report.State != nil {
if report.State.ActiveSlot != activeSlot || report.State.ActiveRelease != report.Observed.ActiveRelease {
add("active_release_drift", "error", "Recorded active state does not match the observed active release pointer.")
}
if report.State.PreviousRelease != report.Observed.PreviousRelease {
add("previous_release_drift", "warning", "Recorded rollback state does not match the observed previous release pointer.")
}
if cfg.Deployment.Strategy == "singleton_candidate" {
leased := report.Observed.CandidateLease && candidateLease != nil
candidateActive := report.Observed.CandidateUnitActive != nil && *report.Observed.CandidateUnitActive
if candidateLease != nil && report.State.CandidateRelease == "" {
add("candidate_lease_without_running_attempt", "error", "An operation-scoped candidate lease remains after the recorded attempt settled.")
}
if candidateLease != nil && report.State.CandidateRelease != "" && candidateLease.Release != report.State.CandidateRelease {
add("candidate_lease_release_mismatch", "error", "The operation-scoped candidate lease does not match the recorded candidate release.")
}
if leased && report.Observed.CandidateUnitActive != nil && !candidateActive {
add("inactive_candidate_lease", "error", "State retains a candidate lease but its operation-scoped unit is inactive.")
}
if !report.Observed.CandidateLease && candidateActive {
add("unleased_candidate_active", "error", "A transient candidate unit is active without a matching running attempt.")
}
if leased && candidateActive && report.Observed.HandlerFileTarget == "candidate" {
add("retained_candidate_routed", "warning", "The retained candidate appears in the handler file; do not stop it before establishing another healthy route.")
}
if leased && candidateActive && report.Observed.HandlerFileTarget != "candidate" {
add("candidate_active_not_routed", "warning", "The leased candidate is active but the handler file does not target it; cleanup remains an explicit reviewed operation.")
}
}
}
switch {
case !report.StateInitialized:
report.Disposition = "state_uninitialized"
case report.Observed.CandidateLease && report.Observed.CandidateUnit == "":
report.Disposition = "legacy_or_invalid_candidate_lease"
case report.Observed.CandidateLease && report.Observed.CandidateUnitActive != nil && *report.Observed.CandidateUnitActive && report.Observed.HandlerFileTarget == "candidate":
report.Disposition = "retained_candidate_handler_file"
case report.Observed.CandidateLease && report.Observed.CandidateUnitActive != nil && *report.Observed.CandidateUnitActive:
report.Disposition = "candidate_active_not_in_handler_file"
case report.Observed.CandidateLease:
report.Disposition = "inactive_candidate_lease"
case len(report.Findings) == 0:
report.Disposition = "settled"
default:
report.Disposition = "manual_review_required"
}
report.Consistent = len(report.Findings) == 0
return report, nil
}
func observeReleaseLink(cfg config.Config, link string, required bool, add func(string, string, string)) string {
release, err := resolveReleaseLink(cfg.Deployment.Root, link)
if err == nil {
return release
}
if !required && os.IsNotExist(err) {
return ""
}
code := "previous_pointer_unreadable"
message := "The configured previous release pointer could not be validated."
severity := "warning"
if required {
code = "active_pointer_unreadable"
message = "The configured active release pointer could not be validated."
severity = "error"
}
add(code, severity, message)
return ""
}
func observeUnit(ctx context.Context, operator Operator, unit string, units map[string]bool, add func(string, string, string)) {
active, err := operator.IsActive(ctx, unit)
if err != nil {
add("unit_unobservable", "error", "A configured service unit state could not be observed.")
return
}
units[unit] = active
}
-182
View File
@@ -1,182 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package deploy
import (
"context"
"os"
"path/filepath"
"strings"
"testing"
"time"
"gamertan.com/tend/internal/config"
"gamertan.com/tend/internal/state"
)
func writeReleaseIdentity(t *testing.T, release, version, commit string) {
t.Helper()
body := `{"version":"` + version + `","commit":"` + commit + `"}`
if err := os.WriteFile(filepath.Join(release, "RELEASE.json"), []byte(body), 0o644); err != nil {
t.Fatal(err)
}
}
func storeSingletonState(t *testing.T, cfg config.Config, active, previous string) {
t.Helper()
record := state.Record{
SchemaVersion: state.SchemaVersion,
Strategy: "singleton_candidate",
DesiredRelease: active,
ActiveSlot: "singleton",
ActiveRelease: active,
PreviousRelease: previous,
LastAttemptRelease: active,
LastAttemptOutcome: "succeeded",
LastAttemptAt: time.Unix(100, 0).UTC().Format(time.RFC3339),
UpdatedAt: time.Unix(100, 0).UTC().Format(time.RFC3339),
}
if err := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, record); err != nil {
t.Fatal(err)
}
}
func reconciliationManager(operator Operator) Manager {
return Manager{Operator: operator, ReadIdentity: readReleaseIdentity}
}
func TestReconcileReportsHealthySingletonWithoutMutation(t *testing.T) {
cfg, active, previous := baseConfig(t, "singleton_candidate")
cfg.Deployment.Singleton, _ = singletonSettings(t, cfg, active)
if err := replaceSymlink(cfg.Deployment.Singleton.PreviousLink, previous); err != nil {
t.Fatal(err)
}
commit := strings.Repeat("a", 40)
writeReleaseIdentity(t, active, "v0.2.0-preview.2", commit)
storeSingletonState(t, cfg, active, previous)
operator := &fakeOperator{active: map[string]bool{
cfg.Deployment.Singleton.Unit: true,
cfg.Service.Name + "-tend-candidate.service": false,
}}
report, err := reconciliationManager(operator).Reconcile(context.Background(), cfg)
if err != nil {
t.Fatal(err)
}
if !report.Consistent || report.Mutation != "none" || !report.StateInitialized || len(report.Findings) != 0 {
t.Fatalf("report=%+v", report)
}
if report.Observed.ActiveRelease != active || report.Observed.PreviousRelease != previous || !report.Observed.RouteHandlerMatches {
t.Fatalf("observed=%+v", report.Observed)
}
if report.Observed.ActiveIdentity == nil || report.Observed.ActiveIdentity.Version != "v0.2.0-preview.2" || report.Observed.ActiveIdentity.Commit != commit {
t.Fatalf("identity=%+v", report.Observed.ActiveIdentity)
}
if !report.Observed.Units[cfg.Deployment.Singleton.Unit] || report.Observed.CandidateUnitActive == nil || *report.Observed.CandidateUnitActive {
t.Fatalf("units=%#v candidate=%v", report.Observed.Units, report.Observed.CandidateUnitActive)
}
}
func TestReconcileReportsHealthyBlueGreenDeployment(t *testing.T) {
cfg, active, previous := baseConfig(t, "blue_green")
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
template := filepath.Join(cfg.Deployment.Root, "handler.template")
if err := os.WriteFile(handler, []byte("reverse_proxy 127.0.0.1:8090\n"), 0o640); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(template, []byte("reverse_proxy {{UPSTREAM}}\n"), 0o644); err != nil {
t.Fatal(err)
}
blue := filepath.Join(cfg.Deployment.Root, "slots", "blue")
green := filepath.Join(cfg.Deployment.Root, "slots", "green")
if err := replaceSymlink(blue, active); err != nil {
t.Fatal(err)
}
if err := replaceSymlink(green, previous); err != nil {
t.Fatal(err)
}
cfg.Deployment.BlueGreen = &config.BlueGreen{
CaddyConfig: filepath.Join(cfg.Deployment.Root, "Caddyfile"), CaddyHandler: handler,
CaddyHandlerTemplate: template, BootstrapActive: "blue",
Blue: config.Slot{Unit: "example-blue.service", Address: "127.0.0.1:8090", Link: blue},
Green: config.Slot{Unit: "example-green.service", Address: "127.0.0.1:8091", Link: green},
}
writeReleaseIdentity(t, active, "v0.2.0-preview.2", strings.Repeat("c", 40))
record := state.Record{
SchemaVersion: state.SchemaVersion, Strategy: "blue_green", DesiredRelease: active,
ActiveSlot: "blue", ActiveRelease: active, PreviousSlot: "green", PreviousRelease: previous,
LastAttemptRelease: active, LastAttemptOutcome: "succeeded",
LastAttemptAt: time.Unix(100, 0).UTC().Format(time.RFC3339), UpdatedAt: time.Unix(100, 0).UTC().Format(time.RFC3339),
}
if err := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, record); err != nil {
t.Fatal(err)
}
operator := &fakeOperator{active: map[string]bool{
"example-blue.service": true, "example-green.service": true,
}}
report, err := reconciliationManager(operator).Reconcile(context.Background(), cfg)
if err != nil {
t.Fatal(err)
}
if !report.Consistent || report.Mutation != "none" || report.Observed.ActiveRelease != active || report.Observed.PreviousRelease != previous || !report.Observed.RouteHandlerMatches {
t.Fatalf("report=%+v", report)
}
if report.Observed.CandidateUnit != "" || report.Observed.CandidateUnitActive != nil {
t.Fatalf("unexpected candidate observation=%+v", report.Observed)
}
}
func TestReconcileExplainsDriftWithoutRepairingIt(t *testing.T) {
cfg, recorded, observed := baseConfig(t, "singleton_candidate")
var handler string
cfg.Deployment.Singleton, handler = singletonSettings(t, cfg, recorded)
if err := replaceSymlink(cfg.Deployment.Singleton.PreviousLink, observed); err != nil {
t.Fatal(err)
}
storeSingletonState(t, cfg, recorded, observed)
if err := replaceSymlink(cfg.Deployment.Singleton.CurrentLink, observed); err != nil {
t.Fatal(err)
}
writeReleaseIdentity(t, observed, "v0.2.0-preview.3", strings.Repeat("b", 40))
candidateHandler, err := renderHandler(cfg.Deployment.Singleton.CaddyHandlerTemplate, cfg.Deployment.Singleton.CandidateAddress)
if err != nil {
t.Fatal(err)
}
if err := os.WriteFile(handler, candidateHandler, 0o640); err != nil {
t.Fatal(err)
}
beforeHandler, err := os.ReadFile(handler)
if err != nil {
t.Fatal(err)
}
operator := &fakeOperator{active: map[string]bool{
cfg.Deployment.Singleton.Unit: false,
cfg.Service.Name + "-tend-candidate.service": true,
}}
report, err := reconciliationManager(operator).Reconcile(context.Background(), cfg)
if err != nil {
t.Fatal(err)
}
if report.Consistent || report.Mutation != "none" {
t.Fatalf("report=%+v", report)
}
codes := map[string]bool{}
for _, finding := range report.Findings {
codes[finding.Code] = true
}
for _, code := range []string{"active_release_drift", "active_unit_inactive", "route_handler_drift", "unleased_candidate_active"} {
if !codes[code] {
t.Fatalf("missing %s in %#v", code, report.Findings)
}
}
target, err := resolveReleaseLink(cfg.Deployment.Root, cfg.Deployment.Singleton.CurrentLink)
if err != nil || target != observed {
t.Fatalf("current=%q err=%v", target, err)
}
afterHandler, err := os.ReadFile(handler)
if err != nil || string(afterHandler) != string(beforeHandler) {
t.Fatalf("handler changed err=%v", err)
}
}
-9
View File
@@ -196,15 +196,6 @@ func extractArtifact(artifact, stage string) error {
_ = out.Close() _ = out.Close()
return err return err
} }
// OpenFile modes are filtered through the caller's umask. Tend is
// commonly invoked by a root account with umask 0077, while release
// binaries must remain executable by their dedicated service users.
// Reapply the validated, name-derived mode explicitly before the file
// becomes part of an immutable release.
if err := out.Chmod(mode); err != nil {
_ = out.Close()
return err
}
if err := out.Sync(); err != nil { if err := out.Sync(); err != nil {
_ = out.Close() _ = out.Close()
return err return err
@@ -1,75 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
//go:build linux
package deploy
import (
"archive/tar"
"compress/gzip"
"os"
"path/filepath"
"syscall"
"testing"
)
func TestExtractArtifactAppliesReleaseModesUnderRestrictiveUmask(t *testing.T) {
dir := t.TempDir()
artifact := filepath.Join(dir, "release.tar.gz")
file, err := os.OpenFile(artifact, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
if err != nil {
t.Fatal(err)
}
gz := gzip.NewWriter(file)
tw := tar.NewWriter(gz)
entries := map[string][]byte{
"bundle/app": []byte("executable"),
"bundle/BUILDINFO.json": []byte("{}"),
"bundle/RELEASE.json": []byte("{}"),
"bundle/SBOM.spdx.json": []byte("{}"),
"bundle/SHA256SUMS": []byte("checksums"),
}
for name, body := range entries {
if err := tw.WriteHeader(&tar.Header{Name: name, Typeflag: tar.TypeReg, Mode: 0o600, Size: int64(len(body))}); err != nil {
t.Fatal(err)
}
if _, err := tw.Write(body); err != nil {
t.Fatal(err)
}
}
if err := tw.Close(); err != nil {
t.Fatal(err)
}
if err := gz.Close(); err != nil {
t.Fatal(err)
}
if err := file.Close(); err != nil {
t.Fatal(err)
}
oldUmask := syscall.Umask(0o077)
t.Cleanup(func() { syscall.Umask(oldUmask) })
stage := filepath.Join(dir, "stage")
if err := os.Mkdir(stage, 0o700); err != nil {
t.Fatal(err)
}
if err := extractArtifact(artifact, stage); err != nil {
t.Fatal(err)
}
for name, want := range map[string]os.FileMode{
"app": 0o755,
"BUILDINFO.json": 0o644,
"RELEASE.json": 0o644,
"SBOM.spdx.json": 0o644,
"SHA256SUMS": 0o644,
} {
info, err := os.Stat(filepath.Join(stage, name))
if err != nil {
t.Fatal(err)
}
if got := info.Mode().Perm(); got != want {
t.Fatalf("%s mode=%#o want=%#o", name, got, want)
}
}
}
-115
View File
@@ -1,115 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package eventlog
import (
"crypto/rand"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"regexp"
"strings"
"syscall"
"time"
)
const Version = 1
var safeValue = regexp.MustCompile(`^[A-Za-z0-9._:/@+-]{1,256}$`)
var hexDigest = regexp.MustCompile(`^[0-9a-f]{64}$`)
var gitCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
var operationID = regexp.MustCompile(`^[0-9a-f]{32}$`)
type Event struct {
Version int `json:"version"`
OperationID string `json:"operation_id"`
Service string `json:"service"`
ArtifactDigest string `json:"artifact_digest"`
Commit string `json:"commit"`
ReleaseVersion string `json:"release_version"`
Phase string `json:"phase"`
Slot string `json:"slot,omitempty"`
DurationMillis int64 `json:"duration_ms"`
Outcome string `json:"outcome"`
ObservedAt string `json:"observed_at"`
}
func OperationID() (string, error) {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
return "", errors.New("cryptographic randomness unavailable")
}
return hex.EncodeToString(b), nil
}
func Append(path string, event Event) error {
if err := event.validate(); err != nil {
return err
}
if !filepath.IsAbs(path) || filepath.Clean(path) != path {
return errors.New("event log path must be absolute and clean")
}
if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
return fmt.Errorf("create event directory: %w", err)
}
if info, err := os.Lstat(path); err == nil {
if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 || info.Mode().Perm()&0o022 != 0 {
return errors.New("event log must be a non-writable regular non-symlink file")
}
} else if !errors.Is(err, os.ErrNotExist) {
return fmt.Errorf("inspect event log: %w", err)
}
b, err := json.Marshal(event)
if err != nil {
return fmt.Errorf("encode deployment event: %w", err)
}
if len(b) > 4096 {
return errors.New("deployment event exceeds bound")
}
b = append(b, '\n')
fd, err := syscall.Open(path, syscall.O_WRONLY|syscall.O_APPEND|syscall.O_CREAT|syscall.O_CLOEXEC|syscall.O_NOFOLLOW, 0o640)
if err != nil {
return fmt.Errorf("open event log: %w", err)
}
file := os.NewFile(uintptr(fd), path)
if file == nil {
_ = syscall.Close(fd)
return errors.New("open event log file")
}
defer file.Close()
n, err := file.Write(b)
if err != nil || n != len(b) {
return errors.New("write complete deployment event")
}
if err := file.Sync(); err != nil {
return fmt.Errorf("sync deployment event: %w", err)
}
return nil
}
func (e Event) validate() error {
if e.Version != Version || !operationID.MatchString(e.OperationID) {
return errors.New("deployment event identity is invalid")
}
if !hexDigest.MatchString(e.ArtifactDigest) || !gitCommit.MatchString(e.Commit) {
return errors.New("deployment event provenance is invalid")
}
for label, value := range map[string]string{"service": e.Service, "artifact_digest": e.ArtifactDigest, "commit": e.Commit, "release_version": e.ReleaseVersion, "phase": e.Phase, "outcome": e.Outcome} {
if !safeValue.MatchString(value) || strings.ContainsRune(value, '\x00') {
return fmt.Errorf("deployment event %s is invalid", label)
}
}
if e.Slot != "" && !safeValue.MatchString(e.Slot) {
return errors.New("deployment event slot is invalid")
}
if e.DurationMillis < 0 {
return errors.New("deployment event duration is invalid")
}
if _, err := time.Parse(time.RFC3339Nano, e.ObservedAt); err != nil {
return errors.New("deployment event timestamp is invalid")
}
return nil
}
-59
View File
@@ -1,59 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package eventlog
import (
"encoding/json"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestAppendBoundedEvent(t *testing.T) {
path := filepath.Join(t.TempDir(), "events.jsonl")
id, err := OperationID()
if err != nil {
t.Fatal(err)
}
event := Event{Version: 1, OperationID: id, Service: "site", ArtifactDigest: strings.Repeat("a", 64), Commit: strings.Repeat("b", 40), ReleaseVersion: "v0.2.0-preview.1", Phase: "activation", Slot: "green", Outcome: "succeeded", ObservedAt: time.Now().UTC().Format(time.RFC3339Nano)}
if err := Append(path, event); err != nil {
t.Fatal(err)
}
b, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
var decoded Event
if err := json.Unmarshal(b, &decoded); err != nil {
t.Fatal(err)
}
if decoded.OperationID != id || strings.Contains(string(b), "secret") {
t.Fatalf("unexpected event: %s", b)
}
if info, _ := os.Stat(path); info.Mode().Perm() != 0o640 {
t.Fatalf("mode=%04o", info.Mode().Perm())
}
}
func TestAppendRejectsUnboundedValuesAndSymlink(t *testing.T) {
id, _ := OperationID()
event := Event{Version: 1, OperationID: id, Service: "site\nsecret", ArtifactDigest: "digest", Commit: "commit", ReleaseVersion: "version", Phase: "activation", Outcome: "failed", ObservedAt: time.Now().UTC().Format(time.RFC3339Nano)}
if err := Append(filepath.Join(t.TempDir(), "events.jsonl"), event); err == nil {
t.Fatal("expected unsafe value rejection")
}
dir := t.TempDir()
target := filepath.Join(dir, "target")
if err := os.WriteFile(target, nil, 0o640); err != nil {
t.Fatal(err)
}
link := filepath.Join(dir, "events.jsonl")
if err := os.Symlink(target, link); err != nil {
t.Skip(err)
}
event.Service = "site"
if err := Append(link, event); err == nil {
t.Fatal("expected symlink rejection")
}
}
-27
View File
@@ -6,7 +6,6 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"path/filepath"
"strconv" "strconv"
"strings" "strings"
@@ -19,17 +18,6 @@ type Source struct {
} }
func Inspect(ctx context.Context, runner process.Runner, dir, branch string) (Source, error) { func Inspect(ctx context.Context, runner process.Runner, dir, branch string) (Source, error) {
gitDir, err := gitPath(ctx, runner, dir, "--git-dir")
if err != nil {
return Source{}, fmt.Errorf("inspect Git directory: %w", err)
}
commonDir, err := gitPath(ctx, runner, dir, "--git-common-dir")
if err != nil {
return Source{}, fmt.Errorf("inspect Git common directory: %w", err)
}
if gitDir != commonDir {
return Source{}, errors.New("release packaging does not yet support linked Git worktrees; use a clean standalone clone of the exact pushed commit")
}
status, err := runner.Run(ctx, dir, nil, "git", "status", "--porcelain=v1", "--untracked-files=all") status, err := runner.Run(ctx, dir, nil, "git", "status", "--porcelain=v1", "--untracked-files=all")
if err != nil { if err != nil {
return Source{}, err return Source{}, err
@@ -63,18 +51,3 @@ func Inspect(ctx context.Context, runner process.Runner, dir, branch string) (So
} }
return Source{Commit: commit, Epoch: epoch}, nil return Source{Commit: commit, Epoch: epoch}, nil
} }
func gitPath(ctx context.Context, runner process.Runner, dir, argument string) (string, error) {
out, err := runner.Run(ctx, dir, nil, "git", "rev-parse", argument)
if err != nil {
return "", err
}
path := strings.TrimSpace(string(out))
if path == "" {
return "", errors.New("Git returned an empty path")
}
if !filepath.IsAbs(path) {
path = filepath.Join(dir, path)
}
return filepath.Clean(path), nil
}
-52
View File
@@ -1,52 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package provenance
import (
"context"
"errors"
"strings"
"testing"
)
type recordingRunner struct {
responses map[string][]byte
calls []string
}
func (runner *recordingRunner) Run(_ context.Context, _ string, _ map[string]string, name string, args ...string) ([]byte, error) {
key := name + " " + strings.Join(args, " ")
runner.calls = append(runner.calls, key)
response, ok := runner.responses[key]
if !ok {
return nil, errors.New("unexpected command: " + key)
}
return response, nil
}
func TestInspectAcceptsStandaloneExactPushedCheckout(t *testing.T) {
commit := strings.Repeat("a", 40)
runner := &recordingRunner{responses: map[string][]byte{
"git rev-parse --git-dir": []byte(".git\n"),
"git rev-parse --git-common-dir": []byte(".git\n"),
"git status --porcelain=v1 --untracked-files=all": nil,
"git rev-parse HEAD": []byte(commit + "\n"),
"git ls-remote --exit-code origin refs/heads/main": []byte(commit + "\trefs/heads/main\n"),
"git show -s --format=%ct " + commit: []byte("1720000000\n"),
}}
result, err := Inspect(context.Background(), runner, "/source", "main")
if err != nil || result.Commit != commit || result.Epoch != 1720000000 {
t.Fatalf("result=%+v err=%v", result, err)
}
}
func TestInspectExplainsUnsupportedLinkedWorktreeBeforeRemoteOrBuildWork(t *testing.T) {
runner := &recordingRunner{responses: map[string][]byte{
"git rev-parse --git-dir": []byte("/repo/.git/worktrees/release\n"),
"git rev-parse --git-common-dir": []byte("/repo/.git\n"),
}}
_, err := Inspect(context.Background(), runner, "/source", "main")
if err == nil || !strings.Contains(err.Error(), "linked Git worktrees") || len(runner.calls) != 2 {
t.Fatalf("calls=%#v err=%v", runner.calls, err)
}
}
-14
View File
@@ -1,14 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
//go:build linux
package serverpolicy
import (
"os"
"syscall"
)
func rootOwned(info os.FileInfo) bool {
stat, ok := info.Sys().(*syscall.Stat_t)
return ok && stat.Uid == 0
}
-8
View File
@@ -1,8 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
//go:build !linux
package serverpolicy
import "os"
func rootOwned(os.FileInfo) bool { return false }
-300
View File
@@ -1,300 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
// Package serverpolicy validates the root-owned allowlist used by Tend's
// restricted SSH receiver. It contains service names and paths, never secrets.
package serverpolicy
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"gamertan.com/tend/internal/config"
)
const SchemaVersion = 1
var servicePattern = regexp.MustCompile(`^[a-z][a-z0-9-]{1,62}$`)
type Policy struct {
SchemaVersion int `json:"schema_version"`
ConfigRoot string `json:"config_root"`
IncomingRoot string `json:"incoming_root"`
SharedLockFile string `json:"shared_lock_file"`
Services map[string]ServicePolicy `json:"services"`
}
type ServicePolicy struct {
Config string `json:"config"`
MaxArtifactBytes int64 `json:"max_artifact_bytes"`
}
type CheckedService struct {
Name string
Config config.Config
Policy ServicePolicy
}
func Parse(reader io.Reader) (Policy, error) {
limited := io.LimitReader(reader, 1<<20+1)
body, err := io.ReadAll(limited)
if err != nil {
return Policy{}, err
}
if len(body) > 1<<20 {
return Policy{}, errors.New("server policy exceeds 1 MiB")
}
decoder := json.NewDecoder(bytes.NewReader(body))
decoder.DisallowUnknownFields()
var policy Policy
if err = decoder.Decode(&policy); err != nil {
return Policy{}, fmt.Errorf("decode server policy: %w", err)
}
var trailing any
if err = decoder.Decode(&trailing); !errors.Is(err, io.EOF) {
return Policy{}, errors.New("server policy contains trailing data")
}
if err = policy.Validate(); err != nil {
return Policy{}, err
}
return policy, nil
}
func Load(path string) (Policy, error) {
if err := secureDirectory(filepath.Dir(path), 0); err != nil {
return Policy{}, fmt.Errorf("server policy directory: %w", err)
}
if err := secureFile(path, 0o600); err != nil {
return Policy{}, fmt.Errorf("server policy: %w", err)
}
file, err := os.Open(path)
if err != nil {
return Policy{}, err
}
defer file.Close()
return Parse(file)
}
func (policy Policy) Validate() error {
if policy.SchemaVersion != SchemaVersion {
return fmt.Errorf("server policy schema_version must be %d", SchemaVersion)
}
if policy.ConfigRoot != "/etc/tend/services" {
return errors.New("server policy config_root must be /etc/tend/services")
}
if policy.IncomingRoot != "/var/lib/tend/incoming" {
return errors.New("server policy incoming_root must be /var/lib/tend/incoming")
}
if policy.SharedLockFile != config.SharedLockFile {
return fmt.Errorf("server policy shared_lock_file must be %s", config.SharedLockFile)
}
if len(policy.Services) == 0 || len(policy.Services) > 128 {
return errors.New("server policy must allow 1 to 128 services")
}
for name, service := range policy.Services {
if !servicePattern.MatchString(name) {
return fmt.Errorf("invalid service name %q", name)
}
expected := filepath.Join(policy.ConfigRoot, name+".json")
if service.Config != expected {
return fmt.Errorf("service %s config must be %s", name, expected)
}
if service.MaxArtifactBytes < 1<<20 || service.MaxArtifactBytes > 512<<20 {
return fmt.Errorf("service %s artifact limit is invalid", name)
}
}
return nil
}
func (policy Policy) CheckFiles() ([]CheckedService, error) {
if err := policy.CheckDirectories(); err != nil {
return nil, err
}
names := make([]string, 0, len(policy.Services))
for name := range policy.Services {
names = append(names, name)
}
sort.Strings(names)
checked := make([]CheckedService, 0, len(names))
for _, name := range names {
service, err := policy.CheckService(name)
if err != nil {
return nil, err
}
checked = append(checked, service)
}
return checked, nil
}
func (policy Policy) CheckDirectories() error {
if err := secureDirectory(policy.ConfigRoot, 0); err != nil {
return fmt.Errorf("config root: %w", err)
}
if err := secureDirectory(policy.IncomingRoot, 0o700); err != nil {
return fmt.Errorf("incoming root: %w", err)
}
if err := secureDirectory("/etc/tend/environment", 0o700); err != nil {
return fmt.Errorf("environment root: %w", err)
}
return nil
}
func (policy Policy) CheckService(name string) (CheckedService, error) {
entry, ok := policy.Services[name]
if !ok {
return CheckedService{}, errors.New("service is not allowed by server policy")
}
if err := secureFile(entry.Config, 0); err != nil {
return CheckedService{}, fmt.Errorf("service %s config: %w", name, err)
}
cfg, err := config.Load(entry.Config)
if err != nil {
return CheckedService{}, fmt.Errorf("service %s config: %w", name, err)
}
if cfg.Service.Name != name {
return CheckedService{}, fmt.Errorf("service %s config identity does not match", name)
}
if cfg.Deployment.LockFile != policy.SharedLockFile {
return CheckedService{}, fmt.Errorf("service %s does not use the host-wide lock", name)
}
if err = secureFile(cfg.Service.EnvironmentFile, 0o600); err != nil {
return CheckedService{}, fmt.Errorf("service %s environment file: %w", name, err)
}
if cfg.Deployment.Singleton != nil {
if err = rejectEnvironmentKey(cfg.Service.EnvironmentFile, cfg.Deployment.Singleton.ListenEnv); err != nil {
return CheckedService{}, fmt.Errorf("service %s environment file: %w", name, err)
}
}
return CheckedService{Name: name, Config: cfg, Policy: entry}, nil
}
func CheckConfig(path string, cfg config.Config) error {
expected := filepath.Join("/etc/tend/services", cfg.Service.Name+".json")
if path != expected {
return fmt.Errorf("production config must be %s", expected)
}
if err := secureDirectory("/etc/tend/services", 0); err != nil {
return fmt.Errorf("config root: %w", err)
}
if err := secureDirectory("/etc/tend/environment", 0o700); err != nil {
return fmt.Errorf("environment root: %w", err)
}
if err := secureFile(path, 0); err != nil {
return fmt.Errorf("production config: %w", err)
}
if cfg.Deployment.LockFile != config.SharedLockFile {
return fmt.Errorf("deployment lock must be %s", config.SharedLockFile)
}
if err := secureFile(cfg.Service.EnvironmentFile, 0o600); err != nil {
return fmt.Errorf("environment file: %w", err)
}
if cfg.Deployment.Singleton != nil {
if err := rejectEnvironmentKey(cfg.Service.EnvironmentFile, cfg.Deployment.Singleton.ListenEnv); err != nil {
return fmt.Errorf("environment file: %w", err)
}
}
return nil
}
func rejectEnvironmentKey(path, key string) error {
file, err := os.Open(path)
if err != nil {
return err
}
defer file.Close()
body, err := io.ReadAll(io.LimitReader(file, 1<<20+1))
if err != nil {
return err
}
if len(body) > 1<<20 {
return errors.New("environment file exceeds 1 MiB")
}
for _, raw := range bytes.Split(body, []byte{'\n'}) {
line := strings.TrimSpace(string(raw))
if !strings.HasPrefix(line, key) {
continue
}
remainder := strings.TrimSpace(strings.TrimPrefix(line, key))
if strings.HasPrefix(remainder, "=") {
return errors.New("singleton candidate listen key must not be set in the shared environment file")
}
}
return nil
}
func secureFile(path string, exactMode os.FileMode) error {
if !filepath.IsAbs(path) || filepath.Clean(path) != path || strings.ContainsRune(path, '\x00') {
return errors.New("path must be clean and absolute")
}
if err := rejectSymlinkAncestors(filepath.Dir(path)); err != nil {
return err
}
info, err := os.Lstat(path)
if err != nil {
return err
}
if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 {
return errors.New("must be a regular non-symlink file")
}
if !rootOwned(info) {
return errors.New("must be owned by root")
}
if exactMode != 0 && info.Mode().Perm() != exactMode {
return fmt.Errorf("mode must be %04o", exactMode)
}
if exactMode == 0 && info.Mode().Perm()&0o022 != 0 {
return errors.New("must not be group- or world-writable")
}
return nil
}
func secureDirectory(path string, exactMode os.FileMode) error {
if !filepath.IsAbs(path) || filepath.Clean(path) != path || strings.ContainsRune(path, '\x00') {
return errors.New("path must be clean and absolute")
}
if err := rejectSymlinkAncestors(filepath.Dir(path)); err != nil {
return err
}
info, err := os.Lstat(path)
if err != nil {
return err
}
if !info.IsDir() || info.Mode()&os.ModeSymlink != 0 {
return errors.New("must be a real directory")
}
if !rootOwned(info) {
return errors.New("must be owned by root")
}
if exactMode != 0 && info.Mode().Perm() != exactMode {
return fmt.Errorf("mode must be %04o", exactMode)
}
if exactMode == 0 && info.Mode().Perm()&0o022 != 0 {
return errors.New("must not be group- or world-writable")
}
return nil
}
func rejectSymlinkAncestors(path string) error {
current := string(filepath.Separator)
for _, part := range strings.Split(strings.TrimPrefix(filepath.Clean(path), string(filepath.Separator)), string(filepath.Separator)) {
if part == "" {
continue
}
current = filepath.Join(current, part)
info, err := os.Lstat(current)
if err != nil {
return err
}
if !info.IsDir() || info.Mode()&os.ModeSymlink != 0 {
return fmt.Errorf("symlink or non-directory ancestor refused: %s", current)
}
}
return nil
}
-59
View File
@@ -1,59 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package serverpolicy
import (
"os"
"path/filepath"
"strings"
"testing"
)
func TestParseAcceptsStrictServiceMap(t *testing.T) {
body := `{"schema_version":1,"config_root":"/etc/tend/services","incoming_root":"/var/lib/tend/incoming","shared_lock_file":"/run/lock/tend-deploy.lock","services":{"example-site":{"config":"/etc/tend/services/example-site.json","max_artifact_bytes":1048576}}}`
policy, err := Parse(strings.NewReader(body))
if err != nil {
t.Fatal(err)
}
if policy.Services["example-site"].MaxArtifactBytes != 1<<20 {
t.Fatalf("policy=%+v", policy)
}
}
func TestRejectEnvironmentKeyProtectsSingletonCandidateOverride(t *testing.T) {
directory := t.TempDir()
path := filepath.Join(directory, "service.env")
for _, test := range []struct {
name string
body string
wantErr bool
}{
{name: "shared values only", body: "APP_SECRET=private\n"},
{name: "exact listen key", body: "EXAMPLE_LISTEN=127.0.0.1:8092\n", wantErr: true},
{name: "spaced listen key", body: " EXAMPLE_LISTEN = 127.0.0.1:8092\n", wantErr: true},
{name: "commented listen key", body: "# EXAMPLE_LISTEN=127.0.0.1:8092\n"},
{name: "longer key", body: "EXAMPLE_LISTENER=safe\n"},
} {
t.Run(test.name, func(t *testing.T) {
if err := os.WriteFile(path, []byte(test.body), 0o600); err != nil {
t.Fatal(err)
}
err := rejectEnvironmentKey(path, "EXAMPLE_LISTEN")
if (err != nil) != test.wantErr {
t.Fatalf("error=%v wantErr=%v", err, test.wantErr)
}
})
}
}
func TestParseRejectsUnknownAndForgedPaths(t *testing.T) {
tests := []string{
`{"schema_version":1,"config_root":"/etc/tend/services","incoming_root":"/var/lib/tend/incoming","shared_lock_file":"/run/lock/tend-deploy.lock","services":{"example-site":{"config":"/tmp/example.json","max_artifact_bytes":1048576}}}`,
`{"schema_version":1,"config_root":"/etc/tend/services","incoming_root":"/var/lib/tend/incoming","shared_lock_file":"/run/lock/tend-deploy.lock","services":{},"surprise":true}`,
}
for _, body := range tests {
if _, err := Parse(strings.NewReader(body)); err == nil {
t.Fatalf("accepted %s", body)
}
}
}
-156
View File
@@ -1,156 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package state
import (
"bytes"
"encoding/json"
"errors"
"fmt"
"io"
"net/netip"
"os"
"path/filepath"
"regexp"
"time"
)
const CandidateLeaseSchemaVersion = 1
var (
leaseOperationPattern = regexp.MustCompile(`^[0-9a-f]{32}$`)
leaseServicePattern = regexp.MustCompile(`^[a-z][a-z0-9-]{1,62}$`)
)
type CandidateLease struct {
SchemaVersion int `json:"schema_version"`
Service string `json:"service"`
OperationID string `json:"operation_id"`
Release string `json:"release"`
Unit string `json:"unit"`
Address string `json:"address"`
StartedAt string `json:"started_at"`
}
func CandidateLeasePath(statePath string) string { return statePath + ".candidate-lease.json" }
func (l CandidateLease) Validate(root, service string) error {
if l.SchemaVersion != CandidateLeaseSchemaVersion {
return errors.New("candidate lease schema version is unsupported")
}
if !leaseServicePattern.MatchString(service) || l.Service != service {
return errors.New("candidate lease service does not match configuration")
}
if !leaseOperationPattern.MatchString(l.OperationID) {
return errors.New("candidate lease operation ID is invalid")
}
if err := releaseBelow(root, l.Release); err != nil {
return fmt.Errorf("candidate lease release: %w", err)
}
expectedUnit := service + "-tend-candidate-" + l.OperationID[:12] + ".service"
if l.Unit != expectedUnit {
return errors.New("candidate lease unit does not match its operation")
}
address, err := netip.ParseAddrPort(l.Address)
if err != nil || !address.Addr().IsLoopback() || address.Port() == 0 {
return errors.New("candidate lease address is invalid")
}
if _, err := time.Parse(time.RFC3339, l.StartedAt); err != nil {
return errors.New("candidate lease timestamp is invalid")
}
return nil
}
func LoadCandidateLease(path, root, service string) (CandidateLease, error) {
info, err := os.Lstat(path)
if err != nil {
return CandidateLease{}, err
}
if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 || info.Size() > 64<<10 {
return CandidateLease{}, errors.New("candidate lease must be a bounded regular file")
}
body, err := os.ReadFile(path)
if err != nil {
return CandidateLease{}, err
}
decoder := json.NewDecoder(bytes.NewReader(body))
decoder.DisallowUnknownFields()
var lease CandidateLease
if err := decoder.Decode(&lease); err != nil {
return CandidateLease{}, fmt.Errorf("decode candidate lease: %w", err)
}
var extra any
if err := decoder.Decode(&extra); !errors.Is(err, io.EOF) {
return CandidateLease{}, errors.New("candidate lease contains trailing data")
}
if err := lease.Validate(root, service); err != nil {
return CandidateLease{}, err
}
return lease, nil
}
func StoreCandidateLease(path, root string, lease CandidateLease) error {
if err := lease.Validate(root, lease.Service); err != nil {
return err
}
if info, err := os.Lstat(path); err == nil && (!info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0) {
return errors.New("candidate lease path must be a regular file, not a symlink")
} else if err != nil && !os.IsNotExist(err) {
return err
}
body, err := json.MarshalIndent(lease, "", " ")
if err != nil {
return err
}
body = append(body, '\n')
dir := filepath.Dir(path)
if err := os.MkdirAll(dir, 0o755); err != nil {
return err
}
temporary, err := os.CreateTemp(dir, ".tend-candidate-lease-")
if err != nil {
return err
}
name := temporary.Name()
complete := false
defer func() {
_ = temporary.Close()
if !complete {
_ = os.Remove(name)
}
}()
if err := temporary.Chmod(0o644); err != nil {
return err
}
if _, err := temporary.Write(body); err != nil {
return err
}
if err := temporary.Sync(); err != nil {
return err
}
if err := temporary.Close(); err != nil {
return err
}
if err := os.Rename(name, path); err != nil {
return err
}
complete = true
return syncDir(dir)
}
func RemoveCandidateLease(path string) error {
info, err := os.Lstat(path)
if os.IsNotExist(err) {
return nil
}
if err != nil {
return err
}
if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 {
return errors.New("candidate lease path must be a regular file, not a symlink")
}
if err := os.Remove(path); err != nil {
return err
}
return syncDir(filepath.Dir(path))
}
-89
View File
@@ -1,89 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package state
import (
"bytes"
"os"
"path/filepath"
"strings"
"testing"
"time"
)
func TestCandidateLeaseRoundTripAndRemoval(t *testing.T) {
root := filepath.Join(t.TempDir(), "service")
release := filepath.Join(root, "releases", "sha256-"+strings.Repeat("a", 64))
if err := os.MkdirAll(release, 0o755); err != nil {
t.Fatal(err)
}
path := CandidateLeasePath(filepath.Join(root, "state.json"))
operation := strings.Repeat("d", 32)
lease := CandidateLease{SchemaVersion: CandidateLeaseSchemaVersion, Service: "example-site", OperationID: operation, Release: release, Unit: "example-site-tend-candidate-" + operation[:12] + ".service", Address: "127.0.0.1:18092", StartedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
if err := StoreCandidateLease(path, root, lease); err != nil {
t.Fatal(err)
}
loaded, err := LoadCandidateLease(path, root, "example-site")
if err != nil || loaded != lease {
t.Fatalf("loaded=%+v err=%v", loaded, err)
}
if err := RemoveCandidateLease(path); err != nil {
t.Fatal(err)
}
if _, err := os.Lstat(path); !os.IsNotExist(err) {
t.Fatalf("lease remains: %v", err)
}
}
func TestCandidateLeaseRejectsCrossServiceAndSymlink(t *testing.T) {
root := filepath.Join(t.TempDir(), "service")
release := filepath.Join(root, "releases", "sha256-"+strings.Repeat("a", 64))
if err := os.MkdirAll(release, 0o755); err != nil {
t.Fatal(err)
}
operation := strings.Repeat("d", 32)
lease := CandidateLease{SchemaVersion: CandidateLeaseSchemaVersion, Service: "example-site", OperationID: operation, Release: release, Unit: "example-site-tend-candidate-" + operation[:12] + ".service", Address: "127.0.0.1:18092", StartedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
if err := lease.Validate(root, "other-site"); err == nil {
t.Fatal("expected service mismatch")
}
path := CandidateLeasePath(filepath.Join(root, "state.json"))
if err := os.Symlink(filepath.Join(root, "elsewhere"), path); err != nil {
t.Fatal(err)
}
if err := StoreCandidateLease(path, root, lease); err == nil {
t.Fatal("expected symlink refusal")
}
if err := RemoveCandidateLease(path); err == nil {
t.Fatal("expected symlink removal refusal")
}
}
func TestCandidateLeasePreservesSchemaOneDeploymentState(t *testing.T) {
root := filepath.Join(t.TempDir(), "service")
release := filepath.Join(root, "releases", "sha256-"+strings.Repeat("a", 64))
if err := os.MkdirAll(release, 0o755); err != nil {
t.Fatal(err)
}
at := time.Unix(1, 0).UTC().Format(time.RFC3339)
statePath := filepath.Join(root, "state.json")
record := Record{SchemaVersion: SchemaVersion, Strategy: "singleton_candidate", DesiredRelease: release, CandidateRelease: release, ActiveSlot: "singleton", ActiveRelease: release, LastAttemptRelease: release, LastAttemptOutcome: "running", LastAttemptAt: at, UpdatedAt: at}
if err := Store(statePath, root, record); err != nil {
t.Fatal(err)
}
operation := strings.Repeat("d", 32)
lease := CandidateLease{SchemaVersion: CandidateLeaseSchemaVersion, Service: "example-site", OperationID: operation, Release: release, Unit: "example-site-tend-candidate-" + operation[:12] + ".service", Address: "127.0.0.1:18092", StartedAt: at}
if err := StoreCandidateLease(CandidateLeasePath(statePath), root, lease); err != nil {
t.Fatal(err)
}
body, err := os.ReadFile(statePath)
if err != nil {
t.Fatal(err)
}
if !bytes.Contains(body, []byte(`"schema_version": 1`)) || bytes.Contains(body, []byte("candidate_operation_id")) || bytes.Contains(body, []byte("candidate_unit")) {
t.Fatalf("deployment state contract changed: %s", body)
}
loaded, err := Load(statePath, root, "singleton_candidate")
if err != nil || loaded.SchemaVersion != 1 || loaded.CandidateRelease != release {
t.Fatalf("loaded=%+v err=%v", loaded, err)
}
}
-25
View File
@@ -19,15 +19,10 @@ const SchemaVersion = 1
type Record struct { type Record struct {
SchemaVersion int `json:"schema_version"` SchemaVersion int `json:"schema_version"`
Strategy string `json:"strategy"` Strategy string `json:"strategy"`
DesiredRelease string `json:"desired_release,omitempty"`
CandidateRelease string `json:"candidate_release,omitempty"`
ActiveSlot string `json:"active_slot"` ActiveSlot string `json:"active_slot"`
ActiveRelease string `json:"active_release"` ActiveRelease string `json:"active_release"`
PreviousSlot string `json:"previous_slot,omitempty"` PreviousSlot string `json:"previous_slot,omitempty"`
PreviousRelease string `json:"previous_release,omitempty"` PreviousRelease string `json:"previous_release,omitempty"`
LastAttemptRelease string `json:"last_attempt_release,omitempty"`
LastAttemptOutcome string `json:"last_attempt_outcome,omitempty"`
LastAttemptAt string `json:"last_attempt_at,omitempty"`
UpdatedAt string `json:"updated_at"` UpdatedAt string `json:"updated_at"`
} }
@@ -73,26 +68,6 @@ func (r Record) Validate(root, strategy string) error {
return fmt.Errorf("previous release: %w", err) return fmt.Errorf("previous release: %w", err)
} }
} }
for label, release := range map[string]string{"desired release": r.DesiredRelease, "candidate release": r.CandidateRelease, "last attempt release": r.LastAttemptRelease} {
if release != "" {
if err := releaseBelow(root, release); err != nil {
return fmt.Errorf("%s: %w", label, err)
}
}
}
if r.LastAttemptOutcome != "" {
switch r.LastAttemptOutcome {
case "running", "succeeded", "failed", "rolled_back":
default:
return errors.New("last attempt outcome is invalid")
}
if _, err := time.Parse(time.RFC3339, r.LastAttemptAt); err != nil {
return errors.New("last attempt timestamp is invalid")
}
}
if r.LastAttemptOutcome == "running" && r.CandidateRelease == "" {
return errors.New("running attempt requires a candidate release")
}
if strategy == "blue_green" && r.PreviousRelease != "" && r.PreviousSlot == r.ActiveSlot { if strategy == "blue_green" && r.PreviousRelease != "" && r.PreviousSlot == r.ActiveSlot {
return errors.New("previous slot must differ from active slot") return errors.New("previous slot must differ from active slot")
} }
+4 -14
View File
@@ -16,8 +16,7 @@ func TestStoreLoadRoundTripAndRejectSymlink(t *testing.T) {
t.Fatal(err) t.Fatal(err)
} }
path := filepath.Join(root, "state.json") path := filepath.Join(root, "state.json")
at := time.Unix(1, 0).UTC().Format(time.RFC3339) record := Record{SchemaVersion: 1, Strategy: "singleton_candidate", ActiveSlot: "singleton", ActiveRelease: release, UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
record := Record{SchemaVersion: SchemaVersion, Strategy: "singleton_candidate", DesiredRelease: release, ActiveSlot: "singleton", ActiveRelease: release, LastAttemptRelease: release, LastAttemptOutcome: "succeeded", LastAttemptAt: at, UpdatedAt: at}
if err := Store(path, root, record); err != nil { if err := Store(path, root, record); err != nil {
t.Fatal(err) t.Fatal(err)
} }
@@ -25,8 +24,8 @@ func TestStoreLoadRoundTripAndRejectSymlink(t *testing.T) {
if err != nil { if err != nil {
t.Fatal(err) t.Fatal(err)
} }
if loaded.ActiveRelease != release || loaded.DesiredRelease != release || loaded.LastAttemptOutcome != "succeeded" { if loaded.ActiveRelease != release {
t.Fatalf("state=%+v", loaded) t.Fatalf("release=%q", loaded.ActiveRelease)
} }
if err := os.Remove(path); err != nil { if err := os.Remove(path); err != nil {
t.Fatal(err) t.Fatal(err)
@@ -38,17 +37,8 @@ func TestStoreLoadRoundTripAndRejectSymlink(t *testing.T) {
t.Fatal("expected symlink refusal") t.Fatal("expected symlink refusal")
} }
} }
func TestRecordRequiresCandidateForRunningAttempt(t *testing.T) {
root := filepath.Join(t.TempDir(), "service")
release := filepath.Join(root, "releases", "sha256-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
record := Record{SchemaVersion: SchemaVersion, Strategy: "singleton_candidate", DesiredRelease: release, ActiveSlot: "singleton", ActiveRelease: release, LastAttemptRelease: release, LastAttemptOutcome: "running", LastAttemptAt: time.Unix(1, 0).UTC().Format(time.RFC3339), UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
if err := record.Validate(root, "singleton_candidate"); err == nil {
t.Fatal("expected missing candidate rejection")
}
}
func TestRecordRejectsReleaseOutsideRoot(t *testing.T) { func TestRecordRejectsReleaseOutsideRoot(t *testing.T) {
record := Record{SchemaVersion: SchemaVersion, Strategy: "singleton_candidate", ActiveSlot: "singleton", ActiveRelease: "/tmp/other/release", UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)} record := Record{SchemaVersion: 1, Strategy: "singleton_candidate", ActiveSlot: "singleton", ActiveRelease: "/tmp/other/release", UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
if err := record.Validate("/opt/example", "singleton_candidate"); err == nil { if err := record.Validate("/opt/example", "singleton_candidate"); err == nil {
t.Fatal("expected path refusal") t.Fatal("expected path refusal")
} }
-123
View File
@@ -1,123 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
// Package transport implements Tend's bounded, versioned deployment stream.
package transport
import (
"bufio"
"bytes"
"crypto/sha256"
"encoding/binary"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"regexp"
"strings"
)
const (
Protocol = "tend-receive-v1"
MaxArtifactBytes = 512 << 20
maxHeaderBytes = 64 << 10
maxArtifactName = 128
)
var (
servicePattern = regexp.MustCompile(`^[a-z][a-z0-9-]{1,62}$`)
artifactPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`)
)
type Header struct {
Protocol string `json:"protocol"`
Service string `json:"service"`
ArtifactName string `json:"artifact_name"`
Size int64 `json:"size"`
SHA256 string `json:"sha256"`
ApprovedSHA256 string `json:"approved_sha256"`
Activate bool `json:"activate"`
}
func (h Header) Validate(maxBytes int64) error {
if h.Protocol != Protocol {
return errors.New("unsupported receive protocol")
}
if !servicePattern.MatchString(h.Service) {
return errors.New("invalid service name")
}
if len(h.ArtifactName) > maxArtifactName || !artifactPattern.MatchString(h.ArtifactName) {
return errors.New("invalid artifact name")
}
if h.Size <= 0 || h.Size > maxBytes {
return errors.New("artifact size exceeds policy")
}
if h.SHA256 != h.ApprovedSHA256 || len(h.SHA256) != 64 || strings.ToLower(h.SHA256) != h.SHA256 {
return errors.New("artifact digest was not explicitly approved")
}
if _, err := hex.DecodeString(h.SHA256); err != nil {
return errors.New("artifact digest is not hexadecimal")
}
return nil
}
func Prefix(header Header) ([]byte, error) {
if err := header.Validate(MaxArtifactBytes); err != nil {
return nil, err
}
body, err := json.Marshal(header)
if err != nil {
return nil, err
}
if len(body) > maxHeaderBytes {
return nil, errors.New("receive header exceeds limit")
}
prefix := make([]byte, 4+len(body))
binary.BigEndian.PutUint32(prefix[:4], uint32(len(body)))
copy(prefix[4:], body)
return prefix, nil
}
func ReadHeader(reader *bufio.Reader) (Header, error) {
var size [4]byte
if _, err := io.ReadFull(reader, size[:]); err != nil {
return Header{}, fmt.Errorf("read receive header length: %w", err)
}
length := binary.BigEndian.Uint32(size[:])
if length == 0 || length > maxHeaderBytes {
return Header{}, errors.New("receive header length is invalid")
}
body := make([]byte, length)
if _, err := io.ReadFull(reader, body); err != nil {
return Header{}, fmt.Errorf("read receive header: %w", err)
}
decoder := json.NewDecoder(bytes.NewReader(body))
decoder.DisallowUnknownFields()
var header Header
if err := decoder.Decode(&header); err != nil {
return Header{}, fmt.Errorf("decode receive header: %w", err)
}
var trailing any
if err := decoder.Decode(&trailing); !errors.Is(err, io.EOF) {
return Header{}, errors.New("receive header contains trailing data")
}
return header, nil
}
func CopyArtifact(destination io.Writer, reader *bufio.Reader, header Header, maxBytes int64) error {
if err := header.Validate(maxBytes); err != nil {
return err
}
hash := sha256.New()
written, err := io.CopyN(io.MultiWriter(destination, hash), reader, header.Size)
if err != nil || written != header.Size {
return errors.New("artifact stream ended before declared size")
}
if _, err = reader.ReadByte(); !errors.Is(err, io.EOF) {
return errors.New("artifact stream contains trailing bytes")
}
if hex.EncodeToString(hash.Sum(nil)) != header.SHA256 {
return errors.New("artifact stream digest does not match")
}
return nil
}
-81
View File
@@ -1,81 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package transport
import (
"bufio"
"bytes"
"crypto/sha256"
"encoding/hex"
"strings"
"testing"
)
func validFrame(t *testing.T, artifact string) ([]byte, Header) {
t.Helper()
hash := sha256.Sum256([]byte(artifact))
digest := hex.EncodeToString(hash[:])
header := Header{Protocol: Protocol, Service: "example-site", ArtifactName: "example.tar.gz", Size: int64(len(artifact)), SHA256: digest, ApprovedSHA256: digest, Activate: true}
prefix, err := Prefix(header)
if err != nil {
t.Fatal(err)
}
return append(prefix, artifact...), header
}
func TestProtocolRoundTrip(t *testing.T) {
frame, expected := validFrame(t, "artifact")
reader := bufio.NewReader(bytes.NewReader(frame))
header, err := ReadHeader(reader)
if err != nil || header != expected {
t.Fatalf("header=%+v err=%v", header, err)
}
var artifact bytes.Buffer
if err = CopyArtifact(&artifact, reader, header, 1<<20); err != nil {
t.Fatal(err)
}
if artifact.String() != "artifact" {
t.Fatalf("artifact=%q", artifact.String())
}
}
func TestProtocolRejectsTrailingAndForgedInputs(t *testing.T) {
frame, header := validFrame(t, "artifact")
reader := bufio.NewReader(bytes.NewReader(append(frame, 'x')))
read, _ := ReadHeader(reader)
if err := CopyArtifact(&bytes.Buffer{}, reader, read, 1<<20); err == nil {
t.Fatal("accepted trailing bytes")
}
header.Service = "../../root"
if _, err := Prefix(header); err == nil {
t.Fatal("accepted forged service")
}
header.Service = "example-site"
header.ApprovedSHA256 = strings.Repeat("0", 64)
if _, err := Prefix(header); err == nil {
t.Fatal("accepted unapproved digest")
}
}
func FuzzProtocolFraming(f *testing.F) {
hash := sha256.Sum256([]byte("artifact"))
digest := hex.EncodeToString(hash[:])
prefix, err := Prefix(Header{Protocol: Protocol, Service: "example-site", ArtifactName: "example.tar.gz", Size: 8, SHA256: digest, ApprovedSHA256: digest})
if err != nil {
f.Fatal(err)
}
frame := append(prefix, []byte("artifact")...)
f.Add(frame)
f.Add([]byte{0, 0, 0, 0})
f.Fuzz(func(t *testing.T, input []byte) {
if len(input) > 2<<20 {
t.Skip()
}
reader := bufio.NewReader(bytes.NewReader(input))
header, err := ReadHeader(reader)
if err != nil {
return
}
_ = CopyArtifact(&bytes.Buffer{}, reader, header, 1<<20)
})
}
-153
View File
@@ -1,153 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package transport
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"os"
"os/exec"
"path/filepath"
"regexp"
"strconv"
"strings"
)
var targetPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*@[A-Za-z0-9][A-Za-z0-9.-]*$`)
type PushOptions struct {
Target string
Port int
KnownHosts string
Identity string
Service string
Artifact string
SHA256 string
ApprovedSHA256 string
Activate bool
}
type SSHRunner interface {
Run(context.Context, string, []string, io.Reader) ([]byte, error)
}
type ExecSSHRunner struct{}
func (ExecSSHRunner) Run(ctx context.Context, name string, args []string, stdin io.Reader) ([]byte, error) {
command := exec.CommandContext(ctx, name, args...)
command.Stdin = stdin
var output, diagnostic boundedBuffer
command.Stdout = &output
command.Stderr = &diagnostic
err := command.Run()
if err != nil {
return output.Bytes(), fmt.Errorf("ssh failed: %w: %s", err, strings.TrimSpace(diagnostic.String()))
}
return output.Bytes(), nil
}
type boundedBuffer struct{ bytes.Buffer }
func (b *boundedBuffer) Write(value []byte) (int, error) {
written := len(value)
remaining := (1 << 20) - b.Len()
if remaining > 0 {
if len(value) > remaining {
value = value[:remaining]
}
_, _ = b.Buffer.Write(value)
}
return written, nil
}
func Push(ctx context.Context, runner SSHRunner, options PushOptions) (json.RawMessage, error) {
if !targetPattern.MatchString(options.Target) || strings.HasPrefix(options.Target, "-") {
return nil, errors.New("target must be user@host without shell syntax")
}
if options.Port < 1 || options.Port > 65535 {
return nil, errors.New("SSH port is invalid")
}
if err := safeClientFile(options.KnownHosts, false); err != nil {
return nil, fmt.Errorf("known-hosts file: %w", err)
}
if options.Identity != "" {
if err := safeClientFile(options.Identity, true); err != nil {
return nil, fmt.Errorf("identity file: %w", err)
}
}
if !filepath.IsAbs(options.Artifact) || filepath.Clean(options.Artifact) != options.Artifact || strings.ContainsAny(options.Artifact, "\x00\r\n\t") {
return nil, errors.New("artifact path must be clean and absolute")
}
artifactInfo, err := os.Lstat(options.Artifact)
if err != nil || !artifactInfo.Mode().IsRegular() || artifactInfo.Mode()&os.ModeSymlink != 0 || artifactInfo.Size() <= 0 || artifactInfo.Size() > MaxArtifactBytes {
return nil, errors.New("artifact must be a bounded regular non-symlink file")
}
artifact, err := os.Open(options.Artifact)
if err != nil {
return nil, err
}
defer artifact.Close()
info, err := artifact.Stat()
if err != nil || !info.Mode().IsRegular() || !os.SameFile(artifactInfo, info) {
return nil, errors.New("artifact must be a bounded regular file")
}
hash := sha256.New()
if _, err = io.Copy(hash, artifact); err != nil {
return nil, err
}
actual := hex.EncodeToString(hash.Sum(nil))
if actual != options.SHA256 || options.SHA256 != options.ApprovedSHA256 {
return nil, errors.New("artifact digest was not explicitly approved")
}
if _, err = artifact.Seek(0, io.SeekStart); err != nil {
return nil, err
}
header := Header{Protocol: Protocol, Service: options.Service, ArtifactName: filepath.Base(options.Artifact), Size: info.Size(), SHA256: options.SHA256, ApprovedSHA256: options.ApprovedSHA256, Activate: options.Activate}
prefix, err := Prefix(header)
if err != nil {
return nil, err
}
args := []string{"-F", os.DevNull, "-T", "-p", strconv.Itoa(options.Port), "-o", "BatchMode=yes", "-o", "ClearAllForwardings=yes", "-o", "ExitOnForwardFailure=yes", "-o", "ForwardAgent=no", "-o", "IdentitiesOnly=yes", "-o", "LogLevel=ERROR", "-o", "PermitLocalCommand=no", "-o", "ProxyCommand=none", "-o", "RequestTTY=no", "-o", "StrictHostKeyChecking=yes", "-o", "UserKnownHostsFile=" + options.KnownHosts}
if options.Identity != "" {
args = append(args, "-i", options.Identity)
}
args = append(args, options.Target, Protocol)
output, err := runner.Run(ctx, "ssh", args, io.MultiReader(bytes.NewReader(prefix), artifact))
if err != nil {
return nil, err
}
if !json.Valid(output) {
return nil, errors.New("receiver returned invalid JSON")
}
return json.RawMessage(output), nil
}
func safeClientFile(path string, private bool) error {
if !filepath.IsAbs(path) || filepath.Clean(path) != path || strings.ContainsAny(path, "\x00\r\n\t") {
return errors.New("path must be clean and absolute")
}
parent, err := os.Lstat(filepath.Dir(path))
if err != nil || !parent.IsDir() || parent.Mode()&os.ModeSymlink != 0 || parent.Mode().Perm()&0o022 != 0 {
return errors.New("parent must be a real directory not writable by group or others")
}
info, err := os.Lstat(path)
if err != nil {
return err
}
if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 {
return errors.New("must be a regular non-symlink file")
}
if info.Mode().Perm()&0o022 != 0 {
return errors.New("must not be group- or world-writable")
}
if private && info.Mode().Perm()&0o077 != 0 {
return errors.New("must not be accessible by group or others")
}
return nil
}
-81
View File
@@ -1,81 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package transport
import (
"bufio"
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"io"
"os"
"path/filepath"
"slices"
"testing"
)
type captureRunner struct {
name string
args []string
input []byte
calls int
}
func (runner *captureRunner) Run(_ context.Context, name string, args []string, input io.Reader) ([]byte, error) {
runner.calls++
runner.name = name
runner.args = append([]string(nil), args...)
runner.input, _ = io.ReadAll(input)
return []byte(`{"validated":true,"mutation":"activated"}`), nil
}
func TestPushUsesPinnedSSHAndExactFrame(t *testing.T) {
dir := t.TempDir()
knownHosts := filepath.Join(dir, "known_hosts")
identity := filepath.Join(dir, "identity")
artifact := filepath.Join(dir, "release.tar.gz")
if err := os.WriteFile(knownHosts, []byte("host key\n"), 0o644); err != nil {
t.Fatal(err)
}
if err := os.WriteFile(identity, []byte("private\n"), 0o600); err != nil {
t.Fatal(err)
}
content := []byte("artifact")
if err := os.WriteFile(artifact, content, 0o600); err != nil {
t.Fatal(err)
}
hash := sha256.Sum256(content)
digest := hex.EncodeToString(hash[:])
runner := &captureRunner{}
result, err := Push(context.Background(), runner, PushOptions{Target: "tend-deploy@example.test", Port: 2222, KnownHosts: knownHosts, Identity: identity, Service: "example-site", Artifact: artifact, SHA256: digest, ApprovedSHA256: digest, Activate: true})
if err != nil {
t.Fatal(err)
}
if !bytes.Contains(result, []byte(`"activated"`)) || runner.calls != 1 || runner.name != "ssh" {
t.Fatalf("result=%s calls=%d name=%q", result, runner.calls, runner.name)
}
if !slices.Contains(runner.args, "ProxyCommand=none") || !slices.Contains(runner.args, "StrictHostKeyChecking=yes") || runner.args[len(runner.args)-1] != Protocol {
t.Fatalf("args=%#v", runner.args)
}
reader := bufio.NewReader(bytes.NewReader(runner.input))
header, err := ReadHeader(reader)
if err != nil {
t.Fatal(err)
}
var copied bytes.Buffer
if err = CopyArtifact(&copied, reader, header, 1<<20); err != nil {
t.Fatal(err)
}
if copied.String() != string(content) || header.Service != "example-site" || !header.Activate {
t.Fatalf("header=%+v body=%q", header, copied.String())
}
}
func TestPushRejectsShellTargetBeforeExecution(t *testing.T) {
runner := &captureRunner{}
_, err := Push(context.Background(), runner, PushOptions{Target: "root@example.test;touch", Port: 22})
if err == nil || runner.calls != 0 {
t.Fatalf("err=%v calls=%d", err, runner.calls)
}
}
-48
View File
@@ -1,48 +0,0 @@
// SPDX-License-Identifier: AGPL-3.0-only
package transport
import (
"bufio"
"context"
"fmt"
"io"
"os"
"gamertan.com/tend/internal/deploy"
"gamertan.com/tend/internal/serverpolicy"
)
func Receive(ctx context.Context, input io.Reader, policy serverpolicy.Policy, manager deploy.Manager) (deploy.Report, error) {
if err := policy.CheckDirectories(); err != nil {
return deploy.Report{}, err
}
reader := bufio.NewReaderSize(input, maxHeaderBytes+4)
header, err := ReadHeader(reader)
if err != nil {
return deploy.Report{}, err
}
service, err := policy.CheckService(header.Service)
if err != nil {
return deploy.Report{}, err
}
if err = header.Validate(service.Policy.MaxArtifactBytes); err != nil {
return deploy.Report{}, err
}
file, err := os.CreateTemp(policy.IncomingRoot, ".tend-receive-"+header.Service+"-")
if err != nil {
return deploy.Report{}, err
}
path := file.Name()
defer os.Remove(path)
if err = file.Chmod(0o600); err == nil {
err = CopyArtifact(file, reader, header, service.Policy.MaxArtifactBytes)
}
if closeErr := file.Close(); err == nil {
err = closeErr
}
if err != nil {
return deploy.Report{}, fmt.Errorf("receive artifact: %w", err)
}
return manager.Deploy(ctx, service.Config, deploy.Request{Artifact: path, SHA256: header.SHA256, ApprovedSHA256: header.ApprovedSHA256, Activate: header.Activate})
}
+5 -10
View File
@@ -1,6 +1,6 @@
{ {
"schema_version": 2, "schema_version": 1,
"service": { "name": "tend", "allowed_host": "localhost", "environment_file": "/etc/tend/environment/tend-release-test.env" }, "service": { "name": "tend", "allowed_host": "localhost" },
"build": { "build": {
"package": "./cmd/tend", "package": "./cmd/tend",
"binary": "tend", "binary": "tend",
@@ -12,25 +12,20 @@
"deployment": { "deployment": {
"strategy": "singleton_candidate", "strategy": "singleton_candidate",
"root": "/opt/tend-release-test", "root": "/opt/tend-release-test",
"lock_file": "/run/lock/tend-deploy.lock", "lock_file": "/run/lock/tend-release-test.lock",
"state_file": "/opt/tend-release-test/tend-state.json", "state_file": "/opt/tend-release-test/tend-state.json",
"event_log": "/opt/tend-release-test/deployment-events.jsonl",
"health_path": "/healthz", "health_path": "/healthz",
"readiness_path": "/readyz", "readiness_path": "/readyz",
"candidate_timeout_seconds": 5, "candidate_timeout_seconds": 5,
"activation_window_seconds": 5,
"smoke": [{ "path": "/", "contains": "Tend" }], "smoke": [{ "path": "/", "contains": "Tend" }],
"public_smoke": [{ "url": "https://example.test/", "contains": "Tend" }],
"singleton": { "singleton": {
"unit": "tend-release-test.service", "unit": "tend-release-test.service",
"address": "127.0.0.1:19090", "address": "127.0.0.1:19090",
"candidate_address": "127.0.0.1:19091", "candidate_address": "127.0.0.1:19091",
"listen_env": "TEND_RELEASE_TEST_LISTEN", "listen_env": "TEND_RELEASE_TEST_LISTEN",
"environment": {},
"current_link": "/opt/tend-release-test/current", "current_link": "/opt/tend-release-test/current",
"previous_link": "/opt/tend-release-test/previous", "previous_link": "/opt/tend-release-test/previous"
"caddy_config": "/etc/caddy/Caddyfile",
"caddy_handler": "/etc/caddy/tend-release-test-handler.caddy",
"caddy_handler_template": "/etc/tend/caddy/tend-release-test.template"
} }
} }
} }
-30
View File
@@ -1,30 +0,0 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: AGPL-3.0-only
set -euo pipefail
root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
allow=$root/scripts/public-snapshot.allow
LC_ALL=C sort -c "$allow"
[[ $(LC_ALL=C sort "$allow" | uniq -d | wc -l) -eq 0 ]]
mapfile -t files <"$allow"
[[ ${#files[@]} -gt 0 ]]
for file in "${files[@]}"; do
[[ -n $file && $file != /* && $file != *..* && $file != .gitea/* && $file != .github/* ]]
git -C "$root" cat-file -e "HEAD:$file"
done
work=$(mktemp -d)
trap 'rm -rf -- "$work"' EXIT
mkdir -m 0700 "$work/tree"
git -C "$root" archive HEAD -- "${files[@]}" | tar -xf - -C "$work/tree"
test ! -e "$work/tree/.git"
test ! -e "$work/tree/.gitea"
test ! -e "$work/tree/.github"
private_pattern='/home/[[:alnum:]_.-]+/|BEGIN (RSA|OPENSSH|EC) PRIVATE KEY|gitea[-_]api[[:alnum:]_.-]*token'
if (cd "$work/tree" && rg -n --hidden --glob '!scripts/export-public.sh' --glob '!scripts/check-public-tree.sh' "$private_pattern" .); then
echo "private material found in public tree" >&2
exit 1
fi
(cd "$work/tree" && ./scripts/check-licenses.sh)
(cd "$work/tree" && GOWORK=off go test -count=1 ./...)
(cd "$work/tree" && GOWORK=off go vet ./...)
(cd "$work/tree" && GOWORK=off CGO_ENABLED=0 go build -buildvcs=false -mod=readonly -trimpath -o "$work/tend" ./cmd/tend)
echo "public tree compiles independently"
+1 -2
View File
@@ -44,8 +44,7 @@ done < <(find "$stage" -type f -print | LC_ALL=C sort)
epoch=$(git -C "$root" show -s --format=%ct "$commit") epoch=$(git -C "$root" show -s --format=%ct "$commit")
printf '{"schema_version":1,"source_commit":"%s","source_tree":"%s","source_date_epoch":%s,"file_count":%d}\n' "$commit" "$tree" "$epoch" "${#files[@]}" >"$stage/PUBLIC-SNAPSHOT.json" printf '{"schema_version":1,"source_commit":"%s","source_tree":"%s","source_date_epoch":%s,"file_count":%d}\n' "$commit" "$tree" "$epoch" "${#files[@]}" >"$stage/PUBLIC-SNAPSHOT.json"
(cd "$stage" && sha256sum PUBLIC-SNAPSHOT.json >PUBLIC-SNAPSHOT.sha256) (cd "$stage" && sha256sum PUBLIC-SNAPSHOT.json >PUBLIC-SNAPSHOT.sha256)
private_pattern='/home/[[:alnum:]_.-]+/|BEGIN (RSA|OPENSSH|EC) PRIVATE KEY|gitea[-_]api[[:alnum:]_.-]*token' if (cd "$stage" && rg -n --hidden --glob '!.git/**' --glob '!PUBLIC-SNAPSHOT.json' --glob '!scripts/export-public.sh' '/home/cole|BEGIN (RSA|OPENSSH|EC) PRIVATE KEY|gitea-api\.token' .); then
if (cd "$stage" && rg -n --hidden --glob '!.git/**' --glob '!PUBLIC-SNAPSHOT.json' --glob '!scripts/export-public.sh' "$private_pattern" .); then
echo "private material found" >&2 echo "private material found" >&2
exit 1 exit 1
fi fi
-36
View File
@@ -8,30 +8,13 @@ SECURITY.md
cmd/tend/main.go cmd/tend/main.go
docs/ARCHITECTURE.md docs/ARCHITECTURE.md
docs/DOGFOOD_EVIDENCE.md docs/DOGFOOD_EVIDENCE.md
docs/DOGFOOD_FRICTION.md
docs/PUBLIC_SNAPSHOT.md docs/PUBLIC_SNAPSHOT.md
docs/SCHEMA_V2_MIGRATION.md
docs/THREAT_MODEL.md docs/THREAT_MODEL.md
docs/WALKTHROUGH.md
examples/LICENSE examples/LICENSE
examples/README.md examples/README.md
examples/blue-green/caddy-handler.template examples/blue-green/caddy-handler.template
examples/blue-green/example-site@.service examples/blue-green/example-site@.service
examples/blue-green/tend.json examples/blue-green/tend.json
examples/local/.env.example
examples/server/authorized_keys.example
examples/server/caddy/docs-site.template
examples/server/caddy/example-site.template
examples/server/environment/docs-site.env.example
examples/server/environment/example-site.env.example
examples/server/example-singleton.service
examples/server/receive-policy.json
examples/server/services/docs-site.json
examples/server/services/example-site.json
examples/server/slots/example-site-blue.env
examples/server/slots/example-site-green.env
examples/server/tend-receive.sudoers
examples/singleton/caddy-handler.template
examples/singleton/tend.json examples/singleton/tend.json
go.mod go.mod
internal/config/config.go internal/config/config.go
@@ -39,42 +22,23 @@ internal/config/config_test.go
internal/deploy/deploy.go internal/deploy/deploy.go
internal/deploy/deploy_test.go internal/deploy/deploy_test.go
internal/deploy/lock_linux.go internal/deploy/lock_linux.go
internal/deploy/lock_linux_test.go
internal/deploy/lock_other.go internal/deploy/lock_other.go
internal/deploy/operator.go internal/deploy/operator.go
internal/deploy/operator_test.go internal/deploy/operator_test.go
internal/deploy/ownership_linux.go internal/deploy/ownership_linux.go
internal/deploy/ownership_other.go internal/deploy/ownership_other.go
internal/deploy/reconcile.go
internal/deploy/reconcile_test.go
internal/deploy/release.go internal/deploy/release.go
internal/deploy/release_mode_linux_test.go
internal/deploy/release_test.go internal/deploy/release_test.go
internal/eventlog/eventlog.go
internal/eventlog/eventlog_test.go
internal/packager/packager.go internal/packager/packager.go
internal/packager/packager_test.go internal/packager/packager_test.go
internal/process/run.go internal/process/run.go
internal/provenance/git.go internal/provenance/git.go
internal/provenance/git_test.go
internal/serverpolicy/ownership_linux.go
internal/serverpolicy/ownership_other.go
internal/serverpolicy/policy.go
internal/serverpolicy/policy_test.go
internal/state/lease.go
internal/state/lease_test.go
internal/state/state.go internal/state/state.go
internal/state/state_test.go internal/state/state_test.go
internal/transport/protocol.go
internal/transport/protocol_test.go
internal/transport/push.go
internal/transport/push_test.go
internal/transport/receive.go
internal/version/version.go internal/version/version.go
internal/version/version_test.go internal/version/version_test.go
release/tend.json release/tend.json
scripts/check-licenses.sh scripts/check-licenses.sh
scripts/check-public-tree.sh
scripts/export-public.sh scripts/export-public.sh
scripts/public-snapshot.allow scripts/public-snapshot.allow
scripts/test-public-snapshot.sh scripts/test-public-snapshot.sh
-6
View File
@@ -4,7 +4,6 @@ set -euo pipefail
root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd) root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
cd "$root" cd "$root"
./scripts/check-licenses.sh ./scripts/check-licenses.sh
./scripts/check-public-tree.sh
go test -count=1 ./... go test -count=1 ./...
go test -race -count=1 ./... go test -race -count=1 ./...
go vet ./... go vet ./...
@@ -15,11 +14,6 @@ GOWORK=off CGO_ENABLED=0 go build -mod=readonly -trimpath -o "$work/tend-2" ./cm
cmp "$work/tend-1" "$work/tend-2" cmp "$work/tend-1" "$work/tend-2"
"$work/tend-1" check --config "$root/examples/blue-green/tend.json" >/dev/null "$work/tend-1" check --config "$root/examples/blue-green/tend.json" >/dev/null
"$work/tend-1" check --config "$root/examples/singleton/tend.json" >/dev/null "$work/tend-1" check --config "$root/examples/singleton/tend.json" >/dev/null
"$work/tend-1" check --config "$root/examples/server/services/example-site.json" >/dev/null
"$work/tend-1" check --config "$root/examples/server/services/docs-site.json" >/dev/null
"$work/tend-1" check --config "$root/release/tend.json" >/dev/null
grep -Fqx 'Defaults:tend-deploy env_keep += "SSH_ORIGINAL_COMMAND"' "$root/examples/server/tend-receive.sudoers"
grep -Fqx 'tend-deploy ALL=(root) NOPASSWD: /usr/local/bin/tend receive --policy /etc/tend/receive-policy.json' "$root/examples/server/tend-receive.sudoers"
[[ $(GOWORK=off go list -m all | wc -l) -eq 1 ]] [[ $(GOWORK=off go list -m all | wc -l) -eq 1 ]]
git diff --check git diff --check
echo "Tend verification passed" echo "Tend verification passed"