Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9d9fc83dd0 | ||
|
|
00d1dd4209 |
@@ -1,3 +1,5 @@
|
|||||||
/bin/
|
/bin/
|
||||||
/dist/
|
/dist/
|
||||||
*.tmp
|
*.tmp
|
||||||
|
.env.local
|
||||||
|
*.env.local
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
{"schema_version":1,"source_commit":"a72903c63e1753f9e6ffbf40453c0830bdfc05c5","source_tree":"295641e67eef5979da76746d8ae271249568263e","source_date_epoch":1786730082,"file_count":44}
|
{"schema_version":1,"source_commit":"8aab3db43f35e6a49aa497f45d73701b13fc9f32","source_tree":"992132ea4703437dc13ffdbb04a077816c02caf9","source_date_epoch":1787049308,"file_count":76}
|
||||||
|
|||||||
@@ -1 +1 @@
|
|||||||
b29d919cefddb06e0c47b453ffd55438944fc92da5732dbb6f98cb509a2c371e PUBLIC-SNAPSHOT.json
|
9c6153ab5b861ec5818591269b2a38a14239f68e89911655aaebec444f54a26d PUBLIC-SNAPSHOT.json
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ Linux, systemd, and Caddy. It packages a clean pushed commit, records exact
|
|||||||
build provenance, activates a health-checked candidate, and keeps rollback
|
build provenance, activates a health-checked candidate, and keeps rollback
|
||||||
state explicit.
|
state explicit.
|
||||||
|
|
||||||
The first preview deliberately supports two quiet deployment shapes:
|
The v0.2 development line supports two quiet deployment shapes:
|
||||||
|
|
||||||
- Caddy-switched blue/green services.
|
- Caddy-switched blue/green services.
|
||||||
- A singleton service with an isolated transient candidate check.
|
- A singleton service with an isolated transient candidate check.
|
||||||
@@ -17,21 +17,62 @@ shell hooks. Application-specific data activation remains application-specific.
|
|||||||
|
|
||||||
```text
|
```text
|
||||||
tend check --config tend.json
|
tend check --config tend.json
|
||||||
tend package --config tend.json --version v0.1.0-preview.2 --out dist
|
tend package --config tend.json --version v0.2.0-preview.2 --out dist
|
||||||
tend deploy --config /etc/example/tend.json --artifact FILE --sha256 HEX --approve-sha256 HEX
|
tend push --target tend-deploy@host --known-hosts FILE --service NAME --artifact FILE --sha256 HEX --approve-sha256 HEX
|
||||||
tend status --config /etc/example/tend.json
|
tend receive --policy /etc/tend/receive-policy.json
|
||||||
tend rollback --config /etc/example/tend.json
|
tend check-server --policy /etc/tend/receive-policy.json
|
||||||
tend prune --config /etc/example/tend.json --keep 3 [--apply]
|
tend deploy --config /etc/tend/services/example-site.json --artifact FILE --sha256 HEX --approve-sha256 HEX
|
||||||
|
tend status --config /etc/tend/services/example-site.json
|
||||||
|
tend rollback --config /etc/tend/services/example-site.json
|
||||||
|
tend prune --config /etc/tend/services/example-site.json --keep 3 [--apply]
|
||||||
```
|
```
|
||||||
|
|
||||||
`deploy`, `rollback`, and applied pruning are target-host operations. Transfer
|
`push` transfers one approved artifact through a pinned OpenSSH connection. A
|
||||||
credentials and production configuration stay outside source repositories.
|
forced, no-shell receiver maps the service name to one root-owned configuration;
|
||||||
Dry-run and digest approval are intentional friction.
|
it accepts no remote path, environment value, URL, or shell fragment. Production
|
||||||
|
hosts receive binaries and evidence, never source or Go dependencies.
|
||||||
|
|
||||||
The first public preview has completed maintenance releases and explicit
|
Schema 2 keeps all services under `/etc/tend/services/`, references a distinct
|
||||||
rollback/reactivation for both Gamertan and the Sandwich Hime website using one
|
root-owned `0600` environment file for each service, and serializes activation
|
||||||
reviewed Tend candidate. See the dated
|
through `/run/lock/tend-deploy.lock`. Builds and transfers remain parallel;
|
||||||
|
only the short Caddy/service activation phase is host-wide. Tend is still a
|
||||||
|
single command, not a daemon.
|
||||||
|
|
||||||
|
Each service also keeps a bounded JSONL deployment-event stream and explicit
|
||||||
|
desired, candidate, active, previous, and last-attempt release identities. The
|
||||||
|
stream contains only fixed provenance and lifecycle fields; Observatory may
|
||||||
|
ingest it later, but an event-write failure never blocks deployment or rollback.
|
||||||
|
After Caddy reload, Tend repeatedly probes the configured canonical HTTPS
|
||||||
|
origins for the activation window. Blue/green deployments simultaneously keep
|
||||||
|
checking the previous slot, restoring the prior handler and inactive-slot state
|
||||||
|
if routed traffic or continuity fails.
|
||||||
|
|
||||||
|
Dry-run and digest approval are intentional friction. See the
|
||||||
|
[schema-2 migration guide](docs/SCHEMA_V2_MIGRATION.md) and the
|
||||||
|
[two-service walkthrough](docs/WALKTHROUGH.md).
|
||||||
|
|
||||||
|
Run `./scripts/verify.sh` on Linux. That required release lane exercises tests,
|
||||||
|
the race detector, vet, deterministic builds, schema-2 examples, and the
|
||||||
|
dependency-free module graph. Tend supports Linux hosts with systemd and Caddy;
|
||||||
|
WSL may be used as a Linux development environment, but native Windows is not a
|
||||||
|
supported execution, deployment, or release-gate platform.
|
||||||
|
|
||||||
|
The v0.1 public preview and immutable `v0.2.0-preview.1` each completed
|
||||||
|
maintenance releases and explicit rollback/reactivation for both Gamertan and
|
||||||
|
the Sandwich Hime website using one reviewed candidate. See the dated
|
||||||
[dogfood evidence](docs/DOGFOOD_EVIDENCE.md) for exact scope and limitations.
|
[dogfood evidence](docs/DOGFOOD_EVIDENCE.md) for exact scope and limitations.
|
||||||
|
|
||||||
|
The additive `v0.2.0-preview.2` candidate keeps that transport and activation
|
||||||
|
contract while adding bounded deployment-event JSONL, routed-origin continuity,
|
||||||
|
rollback annotations, and the operational findings recorded through real
|
||||||
|
dogfooding. Preview 1 remains unchanged. Preview 2 will not be tagged until one
|
||||||
|
identical binary has deployed, rolled back, and reactivated Gamertan, the
|
||||||
|
Sandwich Hime website, and Gamertan Observatory.
|
||||||
|
Operational friction discovered while applying the same contract to new
|
||||||
|
services is tracked separately in the
|
||||||
|
[dogfood friction ledger](docs/DOGFOOD_FRICTION.md). The ledger preserves the
|
||||||
|
fail-closed behavior and records candidate product improvements instead of
|
||||||
|
normalizing application-specific deployment workarounds.
|
||||||
The canonical public repository begins with a sanitized root snapshot rather
|
The canonical public repository begins with a sanitized root snapshot rather
|
||||||
than the private development history.
|
than the private development history.
|
||||||
|
|
||||||
|
|||||||
+13
@@ -19,6 +19,19 @@ The release tag and attached candidate must be built from the final reviewed
|
|||||||
source commit. Documentation-only changes after the recorded campaign require
|
source commit. Documentation-only changes after the recorded campaign require
|
||||||
one final identical-candidate maintenance pass before tagging.
|
one final identical-candidate maintenance pass before tagging.
|
||||||
|
|
||||||
|
`v0.2.0-preview.1` is an immutable, additive release line. It requires schema 2,
|
||||||
|
the restricted `push`/`receive` transport, root-owned environment-file
|
||||||
|
references, host-wide activation serialization, and HTTPS public-origin smoke.
|
||||||
|
Its exact released source remains unchanged.
|
||||||
|
|
||||||
|
`v0.2.0-preview.2` adds bounded deployment-event JSONL, routed-origin
|
||||||
|
activation continuity, rollback annotations, and the reviewed operational
|
||||||
|
friction record. It must preserve every Preview 1 security and release gate.
|
||||||
|
The exact same Preview 2 binary must deploy, roll back, and reactivate
|
||||||
|
Gamertan, the Sandwich Hime website, and Gamertan Observatory before the tag is
|
||||||
|
created. EQL is not part of this generic gate; its SQLite/catalog publication
|
||||||
|
needs a dedicated adapter rather than arbitrary hooks.
|
||||||
|
|
||||||
`v0.1.0-preview.1` is immutable but withdrawn: its source and module checksums
|
`v0.1.0-preview.1` is immutable but withdrawn: its source and module checksums
|
||||||
are valid, while a fresh `go install` reports the development identity because
|
are valid, while a fresh `go install` reports the development identity because
|
||||||
the CLI did not yet adopt the tagged module version from Go build information.
|
the CLI did not yet adopt the tagged module version from Go build information.
|
||||||
|
|||||||
+7
-4
@@ -4,11 +4,14 @@ Report suspected vulnerabilities privately to `security@sandwichhime.com`.
|
|||||||
Please include the affected Tend version, configuration shape, reproduction,
|
Please include the affected Tend version, configuration shape, reproduction,
|
||||||
and expected impact. Do not include production credentials or private logs.
|
and expected impact. Do not include production credentials or private logs.
|
||||||
|
|
||||||
Tend treats source repositories, release artifacts, configuration, handwritten
|
Tend treats reviewed source, pinned toolchains, root-owned server policy and
|
||||||
Caddy templates, and operators as trusted. It treats artifact paths, archives,
|
configuration, handwritten Caddy templates, and operators as trusted. It treats
|
||||||
|
requested service names, protocol frames, artifact paths and bytes, archives,
|
||||||
filesystem state, process output, HTTP responses, and deployment targets as
|
filesystem state, process output, HTTP responses, and deployment targets as
|
||||||
adversarial inputs. It never evaluates configuration as shell code.
|
adversarial inputs. It never evaluates configuration as shell code.
|
||||||
|
|
||||||
The preview is not a sandbox and does not make an untrusted repository safe to
|
The preview is not a sandbox and does not make an untrusted repository safe to
|
||||||
build. Run `tend package` only for reviewed source. Production configuration
|
build. Run `tend package` only for reviewed source. Production configuration,
|
||||||
must be root-owned and kept outside repositories.
|
receive policy, host keys, identities, and `0600` environment files stay outside
|
||||||
|
repositories. A secret value appearing in a Tend report, artifact, state file,
|
||||||
|
or process argument is a security defect and should be reported.
|
||||||
|
|||||||
+98
-13
@@ -19,6 +19,8 @@ import (
|
|||||||
"gamertan.com/tend/internal/deploy"
|
"gamertan.com/tend/internal/deploy"
|
||||||
"gamertan.com/tend/internal/packager"
|
"gamertan.com/tend/internal/packager"
|
||||||
"gamertan.com/tend/internal/process"
|
"gamertan.com/tend/internal/process"
|
||||||
|
"gamertan.com/tend/internal/serverpolicy"
|
||||||
|
"gamertan.com/tend/internal/transport"
|
||||||
"gamertan.com/tend/internal/version"
|
"gamertan.com/tend/internal/version"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -41,6 +43,12 @@ func run() error {
|
|||||||
return packageCommand(ctx, os.Args[2:])
|
return packageCommand(ctx, os.Args[2:])
|
||||||
case "deploy":
|
case "deploy":
|
||||||
return deployCommand(ctx, os.Args[2:])
|
return deployCommand(ctx, os.Args[2:])
|
||||||
|
case "push":
|
||||||
|
return pushCommand(ctx, os.Args[2:])
|
||||||
|
case "receive":
|
||||||
|
return receiveCommand(ctx, os.Args[2:])
|
||||||
|
case "check-server":
|
||||||
|
return checkServerCommand(os.Args[2:])
|
||||||
case "status":
|
case "status":
|
||||||
return statusCommand(ctx, os.Args[2:])
|
return statusCommand(ctx, os.Args[2:])
|
||||||
case "rollback":
|
case "rollback":
|
||||||
@@ -54,7 +62,7 @@ func run() error {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
func usage() error {
|
func usage() error {
|
||||||
return errors.New("usage: tend <check|package|deploy|status|rollback|prune|version> [options]")
|
return errors.New("usage: tend <check|package|push|receive|check-server|deploy|status|rollback|prune|version> [options]")
|
||||||
}
|
}
|
||||||
|
|
||||||
func checkCommand(args []string) error {
|
func checkCommand(args []string) error {
|
||||||
@@ -123,7 +131,7 @@ func deployCommand(ctx context.Context, args []string) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if *activate {
|
if *activate {
|
||||||
if err := requireMutationAuthority(absolute); err != nil {
|
if err := requireMutationAuthority(absolute, cfg); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -137,6 +145,90 @@ func deployCommand(ctx context.Context, args []string) error {
|
|||||||
}
|
}
|
||||||
return writeJSON(report)
|
return writeJSON(report)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func pushCommand(ctx context.Context, args []string) error {
|
||||||
|
set := flag.NewFlagSet("push", flag.ContinueOnError)
|
||||||
|
set.SetOutput(os.Stderr)
|
||||||
|
target := set.String("target", "", "dedicated tend-deploy user and host")
|
||||||
|
port := set.Int("port", 22, "pinned SSH port")
|
||||||
|
knownHosts := set.String("known-hosts", "", "absolute pinned known-hosts file")
|
||||||
|
identity := set.String("identity", "", "optional absolute private key")
|
||||||
|
service := set.String("service", "", "allowed service name")
|
||||||
|
artifact := set.String("artifact", "", "immutable release archive")
|
||||||
|
sha := set.String("sha256", "", "expected artifact SHA-256")
|
||||||
|
approved := set.String("approve-sha256", "", "separately reviewed artifact SHA-256")
|
||||||
|
activate := set.Bool("activate", false, "activate after server validation")
|
||||||
|
if err := set.Parse(args); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if set.NArg() != 0 {
|
||||||
|
return errors.New("push accepts no positional arguments")
|
||||||
|
}
|
||||||
|
artifactAbs, err := filepath.Abs(*artifact)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
result, err := transport.Push(ctx, transport.ExecSSHRunner{}, transport.PushOptions{Target: *target, Port: *port, KnownHosts: *knownHosts, Identity: *identity, Service: *service, Artifact: artifactAbs, SHA256: *sha, ApprovedSHA256: *approved, Activate: *activate})
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return writeJSON(result)
|
||||||
|
}
|
||||||
|
|
||||||
|
func receiveCommand(ctx context.Context, args []string) error {
|
||||||
|
set := flag.NewFlagSet("receive", flag.ContinueOnError)
|
||||||
|
set.SetOutput(os.Stderr)
|
||||||
|
policyPath := set.String("policy", "/etc/tend/receive-policy.json", "root-owned receive policy")
|
||||||
|
if err := set.Parse(args); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if set.NArg() != 0 {
|
||||||
|
return errors.New("receive accepts no positional arguments")
|
||||||
|
}
|
||||||
|
if runtime.GOOS != "linux" || os.Geteuid() != 0 {
|
||||||
|
return errors.New("receive requires Linux root execution")
|
||||||
|
}
|
||||||
|
if os.Getenv("SSH_ORIGINAL_COMMAND") != transport.Protocol {
|
||||||
|
return errors.New("receive refused unexpected SSH command")
|
||||||
|
}
|
||||||
|
policy, err := serverpolicy.Load(*policyPath)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
report, err := transport.Receive(ctx, os.Stdin, policy, newManager())
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
return writeJSON(report)
|
||||||
|
}
|
||||||
|
|
||||||
|
func checkServerCommand(args []string) error {
|
||||||
|
set := flag.NewFlagSet("check-server", flag.ContinueOnError)
|
||||||
|
set.SetOutput(os.Stderr)
|
||||||
|
policyPath := set.String("policy", "/etc/tend/receive-policy.json", "root-owned receive policy")
|
||||||
|
if err := set.Parse(args); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if set.NArg() != 0 {
|
||||||
|
return errors.New("check-server accepts no positional arguments")
|
||||||
|
}
|
||||||
|
if runtime.GOOS != "linux" || os.Geteuid() != 0 {
|
||||||
|
return errors.New("check-server requires Linux root execution")
|
||||||
|
}
|
||||||
|
policy, err := serverpolicy.Load(*policyPath)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
services, err := policy.CheckFiles()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
names := make([]string, 0, len(services))
|
||||||
|
for _, service := range services {
|
||||||
|
names = append(names, service.Name)
|
||||||
|
}
|
||||||
|
return writeJSON(map[string]any{"valid": true, "schema_version": policy.SchemaVersion, "services": names})
|
||||||
|
}
|
||||||
func statusCommand(ctx context.Context, args []string) error {
|
func statusCommand(ctx context.Context, args []string) error {
|
||||||
set := flag.NewFlagSet("status", flag.ContinueOnError)
|
set := flag.NewFlagSet("status", flag.ContinueOnError)
|
||||||
set.SetOutput(os.Stderr)
|
set.SetOutput(os.Stderr)
|
||||||
@@ -175,7 +267,7 @@ func rollbackCommand(ctx context.Context, args []string) error {
|
|||||||
if !*activate {
|
if !*activate {
|
||||||
return writeJSON(map[string]any{"validated": true, "mutation": "none", "message": "repeat with --activate to switch to the recorded previous release"})
|
return writeJSON(map[string]any{"validated": true, "mutation": "none", "message": "repeat with --activate to switch to the recorded previous release"})
|
||||||
}
|
}
|
||||||
if err := requireMutationAuthority(absolute); err != nil {
|
if err := requireMutationAuthority(absolute, cfg); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
record, err := newManager().Rollback(ctx, cfg)
|
record, err := newManager().Rollback(ctx, cfg)
|
||||||
@@ -201,7 +293,7 @@ func pruneCommand(args []string) error {
|
|||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if *apply {
|
if *apply {
|
||||||
if err := requireMutationAuthority(absolute); err != nil {
|
if err := requireMutationAuthority(absolute, cfg); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -225,21 +317,14 @@ func loadConfig(path string) (config.Config, string, error) {
|
|||||||
func newManager() deploy.Manager {
|
func newManager() deploy.Manager {
|
||||||
return deploy.NewManager(deploy.SystemOperator{Runner: process.ExecRunner{}, Timeout: 5 * time.Second})
|
return deploy.NewManager(deploy.SystemOperator{Runner: process.ExecRunner{}, Timeout: 5 * time.Second})
|
||||||
}
|
}
|
||||||
func requireMutationAuthority(configPath string) error {
|
func requireMutationAuthority(configPath string, cfg config.Config) error {
|
||||||
if runtime.GOOS != "linux" {
|
if runtime.GOOS != "linux" {
|
||||||
return errors.New("deployment mutations require Linux")
|
return errors.New("deployment mutations require Linux")
|
||||||
}
|
}
|
||||||
if os.Geteuid() != 0 {
|
if os.Geteuid() != 0 {
|
||||||
return errors.New("deployment mutations require root")
|
return errors.New("deployment mutations require root")
|
||||||
}
|
}
|
||||||
info, err := os.Stat(configPath)
|
return serverpolicy.CheckConfig(configPath, cfg)
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if info.Mode().Perm()&0o022 != 0 {
|
|
||||||
return errors.New("production configuration must not be group- or world-writable")
|
|
||||||
}
|
|
||||||
return nil
|
|
||||||
}
|
}
|
||||||
func writeJSON(value any) error {
|
func writeJSON(value any) error {
|
||||||
encoder := json.NewEncoder(os.Stdout)
|
encoder := json.NewEncoder(os.Stdout)
|
||||||
|
|||||||
+45
-11
@@ -17,12 +17,26 @@ requires byte-identical output. The resulting archive contains only:
|
|||||||
|
|
||||||
The external archive digest is the release identity used by deployment.
|
The external archive digest is the release identity used by deployment.
|
||||||
|
|
||||||
|
## Transport boundary
|
||||||
|
|
||||||
|
`tend push` sends one bounded protocol frame to a dedicated account through a
|
||||||
|
pinned OpenSSH host key. It uses an argument vector, disables config files,
|
||||||
|
forwarding, PTYs, local commands, and proxy commands, and requests exactly
|
||||||
|
`tend-receive-v1`. The account's forced command invokes only the root-owned
|
||||||
|
receiver. The receiver's root-owned policy maps an allowlisted service name to
|
||||||
|
one exact configuration path and size ceiling.
|
||||||
|
|
||||||
|
The target receives no source, repository credential, Go cache, dependency, or
|
||||||
|
arbitrary command. The artifact digest must equal both the produced digest and
|
||||||
|
the separately supplied approved digest before it is staged.
|
||||||
|
|
||||||
## Target-host boundary
|
## Target-host boundary
|
||||||
|
|
||||||
`tend deploy` runs on the Linux target. Transfer and SSH authentication stay
|
Schema-2 configurations live below `/etc/tend/services/`. Secrets live only in
|
||||||
outside Tend v0.1; application-local transfer commands move the artifact and
|
separate `/etc/tend/environment/*.env` files that are root-owned, non-symlink,
|
||||||
the identical Tend binary to a root-controlled staging directory. Tend then
|
and mode `0600`. Configurations contain the path, never the values. Candidate
|
||||||
requires the expected digest to be repeated as a separately approved digest.
|
and installed units read the same file; Tend overrides only the candidate's
|
||||||
|
loopback listen address.
|
||||||
|
|
||||||
Dry-run validation extracts into a temporary directory and does not touch the
|
Dry-run validation extracts into a temporary directory and does not touch the
|
||||||
release tree. Activation acquires the shared lock, installs a content-addressed
|
release tree. Activation acquires the shared lock, installs a content-addressed
|
||||||
@@ -36,11 +50,31 @@ probes it, atomically replaces one imported Caddy handler, validates the full
|
|||||||
Caddy configuration, reloads Caddy, and records the prior active slot.
|
Caddy configuration, reloads Caddy, and records the prior active slot.
|
||||||
|
|
||||||
Singleton mode starts the new release in a hardened transient systemd unit on a
|
Singleton mode starts the new release in a hardened transient systemd unit on a
|
||||||
separate loopback address, probes it, stops the candidate, changes the current
|
separate loopback address and probes it. Tend then validates and atomically
|
||||||
release pointer, and restarts the installed singleton unit.
|
routes the imported Caddy handler to that candidate. While the candidate serves
|
||||||
|
the canonical origin, Tend changes the current release pointer, restarts and
|
||||||
|
probes the installed fixed-address unit, validates Caddy again, and routes back
|
||||||
|
to it. The candidate remains healthy through the activation window and stops
|
||||||
|
only after the handoff succeeds.
|
||||||
|
|
||||||
Any failure before state persistence restores the previously observed Caddy
|
After the local post-activation probes, Tend also checks configured HTTPS public
|
||||||
bytes and/or release pointers. Rollback is a separate explicit command over the
|
origins throughout a bounded activation window. Blue/green mode also probes the
|
||||||
recorded state. It rechecks health and readiness, but deliberately does not
|
previous slot, and singleton mode probes the handoff candidate, for health and
|
||||||
apply the current release's content markers to an older release whose routes
|
readiness throughout that window. Any failure before success is recorded
|
||||||
may differ. Pruning preserves both active and previous releases.
|
restores the previously observed Caddy bytes and/or release pointers. State
|
||||||
|
records desired, candidate, active, previous, and last-attempt
|
||||||
|
release identities, including failed attempts without claiming they became
|
||||||
|
active. Rollback is a separate explicit command
|
||||||
|
over the recorded state. It rechecks local health/readiness and public reachability,
|
||||||
|
but deliberately does not apply a future release's content marker to an older
|
||||||
|
release whose routes may differ. Pruning preserves both active and previous releases.
|
||||||
|
|
||||||
|
## Evidence boundary
|
||||||
|
|
||||||
|
Every attempted activation and explicit rollback emits bounded, versioned JSONL
|
||||||
|
events with an operation ID, service, approved artifact digest, source commit,
|
||||||
|
release version, phase, slot, elapsed duration, and outcome. Values are
|
||||||
|
validated rather than copied from command output. The log contains no
|
||||||
|
environment values, arbitrary process output, HTTP bodies, or secret paths.
|
||||||
|
Identity, entropy, file, and downstream observability failures are deliberately
|
||||||
|
best effort and cannot control Tend's deployment or rollback result.
|
||||||
|
|||||||
+158
-2
@@ -1,8 +1,164 @@
|
|||||||
# Preview dogfood evidence
|
# Preview dogfood evidence
|
||||||
|
|
||||||
This is maintainer-run operational evidence, not an independent audit or a
|
This is maintainer-run operational evidence, not an independent audit or a
|
||||||
general reliability claim. It records the acceptance campaign completed on
|
general reliability claim. It preserves each dated campaign and its limitations
|
||||||
August 14, 2026 before Tend's first public preview.
|
instead of rewriting earlier observations as though later fixes had already
|
||||||
|
existed.
|
||||||
|
|
||||||
|
## Final Preview 2 code-candidate campaign — August 18, 2026
|
||||||
|
|
||||||
|
The final v0.2 Preview 2 implementation candidate completed two explicit
|
||||||
|
rollback-and-reactivation cycles for Gamertan, the Sandwich Hime website, and
|
||||||
|
Gamertan Observatory on the production Linux/systemd/Caddy host. The candidate
|
||||||
|
routed singleton traffic to the already-proven transient process while the
|
||||||
|
fixed-address installed unit restarted, then restored the canonical upstream
|
||||||
|
only after loopback and public-origin validation.
|
||||||
|
|
||||||
|
### Assessed Tend candidate
|
||||||
|
|
||||||
|
- Private implementation source commit:
|
||||||
|
`1fd3b9904c46e817c244196dd5d5a90921ca81a2`.
|
||||||
|
- Version: `v0.2.0-preview.2`.
|
||||||
|
- Linux/amd64 binary SHA-256:
|
||||||
|
`adb4753d4e865775d50d618c999f10dfac9a0de945ccfd9d0b8f2e80d65f4597`.
|
||||||
|
- Gitea release-candidate archive SHA-256:
|
||||||
|
`2bdfee2168cb16883d524cf9703adfa6ed5bc2bf44fbbb896993acc5fe6969ec`.
|
||||||
|
- Toolchain: Go 1.26.6, `CGO_ENABLED=0`, `-trimpath`.
|
||||||
|
- Trusted verification run 286 and release-candidate run 287 passed. The
|
||||||
|
archive's checksums, SPDX SBOM, embedded version, commit, clean VCS state,
|
||||||
|
target, and Go build information were independently rechecked before host
|
||||||
|
installation.
|
||||||
|
|
||||||
|
### Application maintenance artifacts
|
||||||
|
|
||||||
|
- Gamertan archive SHA-256:
|
||||||
|
`4700b075640b8b2fb5c17e0e02cf8d96ee67ceee10fe76d108c8b411983a88aa`.
|
||||||
|
- Sandwich Hime website archive SHA-256:
|
||||||
|
`46b4da41cf6703fb8818e7d25e3a9c13e57cf700f5608b1888c4adb3352e4d38`.
|
||||||
|
- Observatory preview 12 archive SHA-256:
|
||||||
|
`9ef0ddd8ec25d8fb75d6a6887e3ba874df7ebba16d3254f6250fcc646f4fd7f4`.
|
||||||
|
|
||||||
|
Every service ended with the intended current release active and the older
|
||||||
|
release retained as the explicit rollback target. Gamertan returned to its
|
||||||
|
green slot; both singleton services returned to their fixed addresses.
|
||||||
|
Candidate ports and the shared lock were free afterward. Caddy and all five
|
||||||
|
installed application units were active with zero restarts and no failed
|
||||||
|
units.
|
||||||
|
|
||||||
|
### Continuity and deployment evidence
|
||||||
|
|
||||||
|
A seven-minute workstation probe sampled the Gamertan origin, both Sandwich
|
||||||
|
Hime origins, Observatory, and EQL health 1,606 times each throughout the
|
||||||
|
campaign. It observed no HTTP failure status. One simultaneous client-side
|
||||||
|
disconnect affected all five destinations during a reload. A controlled
|
||||||
|
90-second replay therefore observed the same validated no-content-change Caddy
|
||||||
|
reload from both the workstation and an independent Linux host. The Linux host
|
||||||
|
recorded 450 successful responses and zero failures for every origin; the
|
||||||
|
workstation alone repeated one common-mode URL transport error across every
|
||||||
|
destination. Caddy retained the same PID with zero restarts and continued
|
||||||
|
serving unrelated requests. The common-mode workstation event is recorded as
|
||||||
|
an observer-path limitation, not server downtime.
|
||||||
|
|
||||||
|
The authoritative deployment-event files finished at 18 Gamertan events, 14
|
||||||
|
Sandwich Hime events, and 16 Observatory events. Observatory's agent cursor for
|
||||||
|
each stream exactly equalled the corresponding file size, proving complete
|
||||||
|
consumption. The agent and applications reported no warning-or-higher journal
|
||||||
|
entries during the campaign. Representative public routes returned HTTP 200,
|
||||||
|
and the EQL origin remained healthy.
|
||||||
|
|
||||||
|
### Findings closed before this campaign
|
||||||
|
|
||||||
|
Two earlier pre-activation artifacts exposed a mode-restoration defect under a
|
||||||
|
hardened root umask. They failed before route or pointer mutation. Tend now
|
||||||
|
reapplies validated archive modes explicitly and tests extraction under umask
|
||||||
|
`0077`.
|
||||||
|
|
||||||
|
The older singleton strategy briefly exposed an unavailable fixed upstream and
|
||||||
|
produced transient Observatory-agent `502`s. The final candidate's routed
|
||||||
|
handoff removed that failure in repeated production activation and rollback.
|
||||||
|
|
||||||
|
This evidence update changes VCS build metadata but not deployment logic. The
|
||||||
|
release policy therefore still requires one last maintenance pass with the
|
||||||
|
exact evidence-bearing candidate before the signed public tag is created.
|
||||||
|
|
||||||
|
## Restricted multi-service campaign — August 16, 2026
|
||||||
|
|
||||||
|
Tend's v0.2 implementation candidate completed the same maintenance contract
|
||||||
|
through the restricted transport for two independently configured services on
|
||||||
|
one Linux/systemd/Caddy host.
|
||||||
|
|
||||||
|
### Assessed Tend implementation
|
||||||
|
|
||||||
|
- Implementation source commit:
|
||||||
|
`840b77da708bbcd87a3203fb6a1f99b2984b8667`.
|
||||||
|
- Linux/amd64 candidate binary SHA-256:
|
||||||
|
`b3961315288871fa6085bf3b75c784a825a88f2bc3dd0694ad5d6c590eddf895`.
|
||||||
|
- Candidate archive SHA-256:
|
||||||
|
`948bad5271dca08b9445c387c5aea7f58f22add6b47b9071e1801a756aa71259`.
|
||||||
|
- Toolchain: Go 1.26.6, `CGO_ENABLED=0`, `-trimpath`.
|
||||||
|
- Trusted Gitea verification run 119 and release-candidate run 120 passed for
|
||||||
|
the exact implementation commit. CI and an independent, network-disabled
|
||||||
|
build each produced the same archive digest twice.
|
||||||
|
- The installed receiver accepted only a pinned Ed25519 host key, a dedicated
|
||||||
|
forced-command deployment key, the exact `tend-receive-v1` protocol, two
|
||||||
|
allowlisted service names, and separately repeated artifact digests. An
|
||||||
|
attempted arbitrary SSH command was refused.
|
||||||
|
- Production source, Go caches, repository credentials, secret values, remote
|
||||||
|
paths, and shell commands did not cross the transport boundary.
|
||||||
|
|
||||||
|
Adding this evidence changes VCS build metadata but not executable logic. The
|
||||||
|
signed preview tag and attached release evidence therefore identify the final
|
||||||
|
evidence-bearing candidate and its required last maintenance pass.
|
||||||
|
|
||||||
|
### Gamertan blue-green campaign
|
||||||
|
|
||||||
|
- Application source commit:
|
||||||
|
`a7e54047d3dc11671824b6ecc8ed698a9dd04421`.
|
||||||
|
- Preview 27 artifact SHA-256:
|
||||||
|
`edadef3a97c089771e2b6bb7dadd58928762284878f76f23b112c61ae282f17f`.
|
||||||
|
- The artifact was built twice, byte-identically, in the pinned Go 1.26.6
|
||||||
|
image with networking disabled and the audited cached module graph.
|
||||||
|
- The inactive slot passed health, readiness, page-marker, Caddy validation,
|
||||||
|
public-origin, Sandwich Hime mount, and EQL continuity checks before traffic
|
||||||
|
moved.
|
||||||
|
- Explicit rollback restored preview 26; its readiness and public boundary
|
||||||
|
passed; preview 27 was then reactivated through the recorded state.
|
||||||
|
- Sandwich Hime's release pointer and service remained unchanged throughout.
|
||||||
|
|
||||||
|
### Sandwich Hime singleton campaign
|
||||||
|
|
||||||
|
- Application source commit:
|
||||||
|
`435880c6751b773b6c5ee3ae6833d26e8eb7c0df`.
|
||||||
|
- Preview 30 artifact SHA-256:
|
||||||
|
`bee8d5bcde2c3c2e8bb96d5909a19889fc7f0be9390046c243fe20c9ff2ca44b`.
|
||||||
|
- The artifact was built twice, byte-identically, in the same pinned,
|
||||||
|
network-disabled Go 1.26.6 environment.
|
||||||
|
- A transient DynamicUser candidate passed health, readiness, tutorial marker,
|
||||||
|
canonical-origin, and Gamertan-mounted checks before the singleton pointer
|
||||||
|
and installed service changed. The candidate port was released afterward.
|
||||||
|
- Explicit rollback restored preview 28; its readiness and public boundary
|
||||||
|
passed; preview 30 was then reactivated through the recorded state.
|
||||||
|
- Both Gamertan slots, its selected Caddy upstream, and EQL remained healthy.
|
||||||
|
|
||||||
|
### Findings resolved by dogfood
|
||||||
|
|
||||||
|
The first restricted transfer stopped before artifact validation because
|
||||||
|
`sudo` removed `SSH_ORIGINAL_COMMAND`. The forced account still refused the
|
||||||
|
request; no service state changed. The 0BSD sudoers template now preserves only
|
||||||
|
that one server-supplied variable, while the receiver requires its exact
|
||||||
|
protocol value and rejects every other command.
|
||||||
|
|
||||||
|
The first singleton candidate stopped before pointer mutation because the
|
||||||
|
shared environment file's live listen value overrode the candidate address.
|
||||||
|
The site remained on its former release and retained the same process. Tend now
|
||||||
|
rejects singleton shared environment files containing the configured listen
|
||||||
|
key; installed units own the non-secret live address and Tend supplies only the
|
||||||
|
transient candidate address.
|
||||||
|
|
||||||
|
Final verification found both service states valid, every installed unit and
|
||||||
|
Caddy active with zero restart failures, the candidate port closed, no warning
|
||||||
|
or error entries after the successful campaign, and representative Gamertan,
|
||||||
|
Sandwich Hime, mounted, and EQL routes returning HTTP 200.
|
||||||
|
|
||||||
## Assessed candidate
|
## Assessed candidate
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,328 @@
|
|||||||
|
# Dogfood friction ledger
|
||||||
|
|
||||||
|
This document records friction observed through August 18, 2026 while using
|
||||||
|
Tend for real maintenance releases. Friction is evidence about the product: it
|
||||||
|
should become either a clearer contract, safer automation, or an explicit
|
||||||
|
non-goal. It must not become application-specific shell lore.
|
||||||
|
|
||||||
|
The findings below are maintainer observations, not implemented promises. Tend
|
||||||
|
continues to fail closed while an option is being designed. In particular,
|
||||||
|
there is no `--skip-remote` packaging escape hatch, no arbitrary deployment
|
||||||
|
hook, and no relaxation of the host-wide activation lock.
|
||||||
|
|
||||||
|
## Design standard
|
||||||
|
|
||||||
|
A Tend workflow should make the secure path the short, documented path:
|
||||||
|
|
||||||
|
- source, artifact, approval, configuration, and active-release identities are
|
||||||
|
explicit and inspectable;
|
||||||
|
- credentials are narrowly scoped, briefly available, and absent from
|
||||||
|
artifacts and logs;
|
||||||
|
- application validation is declarative, bounded, and cannot become a shell;
|
||||||
|
- first installation, maintenance, activation, rollback, and pruning are
|
||||||
|
distinct operations;
|
||||||
|
- every mutation has a recorded prior state and a tested restoration path;
|
||||||
|
- unrelated services may build and prepare candidates concurrently, while the
|
||||||
|
shared Caddy activation boundary remains serialized.
|
||||||
|
|
||||||
|
## Observed findings
|
||||||
|
|
||||||
|
| ID | Status | Finding | Current safe behavior | Candidate direction |
|
||||||
|
| --- | --- | --- | --- | --- |
|
||||||
|
| F-01 | Workflow mitigation | CI packaging needs proof that the exact commit was pushed. | Packaging verifies the configured remote and fails if Git cannot authenticate. The trusted workflow retains only its job-scoped read-only checkout credential. | Make the proof credential or an authenticated source attestation an explicit Tend input and evidence boundary. |
|
||||||
|
| F-02 | Open | Application-owned unit and configuration changes are outside the binary activation transaction. | Operators stage, validate, back up, and restore those files separately. | Add an allowlisted configuration transaction or record and validate exact configuration digests. |
|
||||||
|
| F-03 | Open | A singleton candidate may not reproduce the installed unit's arguments and application configuration. | It receives the production environment file and an isolated listen override; activation still fails closed. | Add a bounded application preflight and explicit, validated candidate invocation. |
|
||||||
|
| F-04 | Partially resolved | Artifact production, review, approval, transfer, and activation require several identity checks. | v0.2 standardizes the evidence bundle and restricted transfer; the operator still repeats the approved digest intentionally. | Add offline inspection and approval ergonomics without combining build authority and production authority. |
|
||||||
|
| F-05 | Partially resolved | Tend maintenance assumes an existing adopted service and current release. | `check-server` validates a prepared host, while first installation remains a separately reviewed operator procedure. | Define an explicit `install` or `adopt` transaction rather than silently treating bootstrap as maintenance. |
|
||||||
|
| F-06 | Partially resolved | Binary rollback can be unsafe when service configuration has changed incompatibly. | v0.2 records desired, candidate, active, previous, and last-attempt releases, but external configuration compatibility is operator-owned. | Bind non-secret configuration identities and preflight results to release state. |
|
||||||
|
| F-07 | Open | `GOPROXY=off` does not prove that every module metadata lookup is available locally. | Packaging stops before artifact creation when Go cannot resolve the complete pinned module graph. | Separate checksum-verified dependency resolution from a network-disabled, cache-completeness-checked build stage. |
|
||||||
|
| F-08 | Open | Packaging an otherwise clean pushed commit from a linked Git worktree fails Go's required VCS-status stamp. | Tend stops before artifact creation; package the same exact commit from a clean standalone clone. | Detect linked worktrees during `check`/`package`, explain the supported source shape, and assess a provenance-preserving worktree build that does not weaken `-buildvcs=true`. |
|
||||||
|
| F-09 | Resolved in Preview 2 | Restarting a fixed-address singleton briefly exposed Caddy to an unavailable upstream. | Tend now routes the imported handler to the candidate, restarts and proves the installed unit behind that handoff, then routes back while the candidate remains healthy through the activation window. | Preserve the production regression campaign and failure-injection matrix. |
|
||||||
|
| F-10 | Resolved in Preview 2 | A hardened root umask could make an extracted application binary executable only by root. | Tend reapplies every validated archive mode explicitly; extraction is tested under umask `0077`. | Preserve the mode and non-root candidate tests. |
|
||||||
|
| F-11 | Evidence practice | A single external observer can report common-mode client or network errors as apparent multi-service downtime. | Production campaigns retain server state and use an independent observer before classifying a continuity failure. | Standardize multi-vantage continuity evidence without making an observability dependency part of deployment authority. |
|
||||||
|
|
||||||
|
## F-01: pushed-commit proof in CI
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
During an Observatory release-candidate run, verification intentionally checked
|
||||||
|
out source without persisted credentials. Tend later attempted to prove that
|
||||||
|
`HEAD` existed on the configured private Gitea branch and Git could not
|
||||||
|
authenticate. Packaging stopped before producing an artifact. This was the
|
||||||
|
correct fail-closed outcome, but the credential lifecycle was not obvious from
|
||||||
|
the workflow contract.
|
||||||
|
|
||||||
|
The trusted workflow uses a job-scoped read-only checkout credential so Tend
|
||||||
|
can perform the independent remote proof. It is not a long-lived repository or
|
||||||
|
deployment credential, and it is removed by checkout cleanup at the end of the
|
||||||
|
job. Production credentials remain unavailable to the build job.
|
||||||
|
|
||||||
|
### Options to assess
|
||||||
|
|
||||||
|
1. Support an explicit read-only Git credential file or credential helper for
|
||||||
|
package-time proof. Never accept a token in arguments, configuration,
|
||||||
|
manifests, artifacts, or logs.
|
||||||
|
2. Accept an authenticated CI source-attestation document binding repository,
|
||||||
|
branch, commit, tree, workflow identity, and event identity. Define which
|
||||||
|
CI issuers are trusted and preserve the attestation with release evidence.
|
||||||
|
3. Accept a signed, pre-verified source bundle whose identity and policy can be
|
||||||
|
checked without network access.
|
||||||
|
|
||||||
|
Do not add a generic skip flag. An unavailable proof must remain a packaging
|
||||||
|
failure unless an equally strong proof mode was selected explicitly.
|
||||||
|
|
||||||
|
## F-02: application configuration is not binary activation
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
Tend can transact an immutable binary, release pointers, a systemd restart, a
|
||||||
|
validated Caddy handler, health checks, and rollback state. It does not
|
||||||
|
currently transact application configuration, systemd unit changes, credential
|
||||||
|
bindings, or server-local secret files. A first deployment that changes these
|
||||||
|
files therefore needs a separate backup, validation, installation, and
|
||||||
|
restoration procedure.
|
||||||
|
|
||||||
|
This boundary is safe but easy to overlook: restoring the previous binary does
|
||||||
|
not restore an incompatible unit or application configuration.
|
||||||
|
|
||||||
|
### Options to assess
|
||||||
|
|
||||||
|
1. Add strict managed-file entries with exact source and destination paths,
|
||||||
|
content digest, owner, group, mode, and validation type. Permit only regular
|
||||||
|
files beneath configured roots; reject symlinks and unknown destinations.
|
||||||
|
2. Add a separate `tend configure` transaction that backs up, atomically
|
||||||
|
replaces, validates, and restores supported systemd/Caddy/application files.
|
||||||
|
3. Keep configuration externally managed, but require Tend to record desired
|
||||||
|
and active configuration digests and prove candidate/rollback compatibility.
|
||||||
|
|
||||||
|
None of these options should permit arbitrary shell commands. Secret values
|
||||||
|
remain referenced server-side and must never enter release artifacts or state.
|
||||||
|
|
||||||
|
## F-03: candidate fidelity
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
A hardened singleton candidate receives the configured production environment
|
||||||
|
file and Tend's isolated listen-address override. It does not currently model
|
||||||
|
the installed systemd unit's complete argument vector. An application whose
|
||||||
|
configuration path is supplied by unit arguments may therefore start a
|
||||||
|
candidate with defaults instead of the intended production configuration.
|
||||||
|
That preserves live-state isolation, but it may prove only executable startup.
|
||||||
|
A missing credential, incompatible configuration field, filesystem permission,
|
||||||
|
or data migration requirement can then surface at activation time.
|
||||||
|
|
||||||
|
### Options to assess
|
||||||
|
|
||||||
|
- Define a fixed application preflight command or protocol that validates the
|
||||||
|
production configuration, credentials, permissions, and data compatibility
|
||||||
|
without binding the live port or mutating live state.
|
||||||
|
- Add a strict candidate argument vector to schema 2. Validate each argument,
|
||||||
|
reject secret values and paths outside the application contract, and pass it
|
||||||
|
directly to systemd without a shell.
|
||||||
|
- Permit an allowlisted `check` argument vector, never a shell string, with
|
||||||
|
bounded time/output and an explicitly non-mutating application contract.
|
||||||
|
- Record which checks ran against the binary alone and which ran against the
|
||||||
|
actual production configuration so the evidence cannot overstate coverage.
|
||||||
|
|
||||||
|
## F-04: artifact review and approval ergonomics
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
The secure flow deliberately separates build, review, digest approval,
|
||||||
|
transfer, candidate validation, and activation. v0.2 now produces a consistent
|
||||||
|
archive, manifest, SBOM, and checksum set, then transfers one exact artifact
|
||||||
|
through its restricted receiver. In practice, operators still need a
|
||||||
|
predictable way to discover that CI artifact, inspect it, approve exactly one
|
||||||
|
digest, and retain the review evidence. Without a first-class review path,
|
||||||
|
correct manual steps are easy to reconstruct differently for each application.
|
||||||
|
|
||||||
|
### Options to assess
|
||||||
|
|
||||||
|
- Add `tend inspect` for offline, non-mutating verification and a concise human
|
||||||
|
and JSON summary of source, build, dependency, and archive identities.
|
||||||
|
- Add an approval record that binds the artifact digest, service, target,
|
||||||
|
approver, and expiry without containing a credential.
|
||||||
|
- Standardize one Gitea artifact layout and documented download-to-activation
|
||||||
|
workflow. Keep production credentials unavailable to verification jobs.
|
||||||
|
|
||||||
|
Approval must remain explicit; better ergonomics must not turn a successful
|
||||||
|
build into an automatic production mutation.
|
||||||
|
|
||||||
|
## F-05: first installation and adoption
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
The maintenance workflow expects an installed service, valid configuration,
|
||||||
|
and a current release pointer. `check-server` validates the restricted receiver
|
||||||
|
policy and prepared service boundary, but it does not create them. Observatory
|
||||||
|
bootstrap therefore required operator-managed service account, directories,
|
||||||
|
credentials, unit, configuration, and an initial current release before Tend
|
||||||
|
could own later maintenance safely.
|
||||||
|
|
||||||
|
### Options to assess
|
||||||
|
|
||||||
|
- `tend install`: a deliberately broader, separately approved transaction with
|
||||||
|
a strict schema and complete rollback of every supported created object.
|
||||||
|
- `tend adopt`: validate an existing service and release, copy or identify its
|
||||||
|
immutable artifact, establish state, and refuse ambiguous ownership.
|
||||||
|
- Keep bootstrap out of Tend, but ship a versioned acceptance checklist and a
|
||||||
|
machine-readable `check-server` result that maintenance can require.
|
||||||
|
|
||||||
|
Installation and adoption must not be inferred from a missing state file.
|
||||||
|
|
||||||
|
## F-06: release and configuration identity
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
Content-addressed releases and v0.2 state make desired, candidate, active,
|
||||||
|
previous, and last-attempt binary identities clear. A live service is still a
|
||||||
|
combination of its binary, application configuration, credential bindings,
|
||||||
|
unit, routing fragment, and sometimes data schema. Tend cannot yet fully
|
||||||
|
explain that combined identity or determine whether a retained binary is
|
||||||
|
compatible with the current external configuration.
|
||||||
|
|
||||||
|
### Options to assess
|
||||||
|
|
||||||
|
- Extend the existing desired, candidate, active, previous, and last-attempt
|
||||||
|
release state with non-secret configuration and unit digests.
|
||||||
|
- Require rollback compatibility declarations or read-only application
|
||||||
|
preflight before changing traffic.
|
||||||
|
- Expose the identities and last validation results through
|
||||||
|
`tend status --json` for deployment evidence and future Observatory
|
||||||
|
ingestion.
|
||||||
|
|
||||||
|
Configuration records contain digests and approved metadata only—not secret
|
||||||
|
contents.
|
||||||
|
|
||||||
|
## F-07: offline module-cache completeness
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
An exact Observatory package attempt used `GOPROXY=off` and a previously used
|
||||||
|
module cache. Source archives for the application dependencies were present,
|
||||||
|
but Go still needed several module metadata records while Tend enumerated the
|
||||||
|
complete build graph. Go refused the lookup and Tend stopped before building
|
||||||
|
or writing an artifact. Re-enabling the checksum-verified public proxy supplied
|
||||||
|
the missing metadata; the resulting package was byte-identical to the trusted
|
||||||
|
CI candidate.
|
||||||
|
|
||||||
|
This is safe failure, but `GOPROXY=off` alone is not evidence of a hermetic
|
||||||
|
build. A cache can be partially populated even when ordinary builds happen to
|
||||||
|
succeed.
|
||||||
|
|
||||||
|
### Options to assess
|
||||||
|
|
||||||
|
- Add a resolver stage that runs with the pinned toolchain, proxy, and checksum
|
||||||
|
database, emits the complete module inventory, and materializes a bounded,
|
||||||
|
read-only cache for the builder.
|
||||||
|
- Run Tend's package stage with networking disabled and require every module,
|
||||||
|
checksum, source archive, and metadata record to come from that reviewed
|
||||||
|
cache.
|
||||||
|
- Add a non-mutating cache-completeness check that reports missing module
|
||||||
|
identities before the expensive double build.
|
||||||
|
- Consider a strictly verified vendored-source mode where repository size and
|
||||||
|
update review are acceptable; do not silently change dependency modes.
|
||||||
|
|
||||||
|
Do not treat `GONOSUMDB`, `GOPRIVATE`, or a proxy bypass as an offline-build
|
||||||
|
solution. They alter verification or routing policy rather than proving cache
|
||||||
|
completeness.
|
||||||
|
|
||||||
|
## F-08: linked-worktree VCS stamping
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
An exact clean, pushed Observatory main commit passed verification and public
|
||||||
|
snapshot isolation from a detached linked Git worktree. Tend accepted its
|
||||||
|
source and remote provenance, then Go 1.26.6 stopped both `tend package` and an
|
||||||
|
equivalent direct build at the required `-buildvcs=true` step with `error
|
||||||
|
obtaining VCS status: exit status 128`. Ordinary Git status and commit queries
|
||||||
|
from the same worktree succeeded.
|
||||||
|
|
||||||
|
A fresh standalone SSH clone of the identical commit packaged successfully
|
||||||
|
twice. Both archives were byte-identical and carried the expected commit, Go
|
||||||
|
version, VCS settings, checksums, manifest, and SBOM. This isolates the failure
|
||||||
|
to the linked-worktree build shape rather than the application source or module
|
||||||
|
graph.
|
||||||
|
|
||||||
|
The failure is safe: Tend did not create a partial artifact and must not switch
|
||||||
|
to `-buildvcs=false`, because the package gate independently verifies the
|
||||||
|
embedded VCS revision and clean state. Until the interaction is resolved, use
|
||||||
|
a clean standalone clone for release packaging.
|
||||||
|
|
||||||
|
### Options to assess
|
||||||
|
|
||||||
|
- Detect a `.git` indirection file during `tend check` and fail before the
|
||||||
|
expensive double build with a precise standalone-clone instruction.
|
||||||
|
- Reproduce the interaction in a package integration test against the minimum
|
||||||
|
supported Go release and determine whether it is a Go toolchain limitation
|
||||||
|
or an invocation/environment defect.
|
||||||
|
- If linked worktrees can be supported, require the resulting build record to
|
||||||
|
carry the exact expected `vcs.revision` and `vcs.modified=false`; do not
|
||||||
|
synthesize those settings or disable VCS stamping.
|
||||||
|
- Consider an explicit, signed source-bundle input as part of the broader
|
||||||
|
source-attestation design. It must remain at least as strong as current
|
||||||
|
pushed-commit proof.
|
||||||
|
|
||||||
|
## F-09: singleton traffic continuity
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
During the August 18 Observatory maintenance exercise, the transient candidate
|
||||||
|
passed health, readiness, and content checks. Tend then stopped that candidate,
|
||||||
|
changed the singleton pointer, and restarted the installed fixed-address unit.
|
||||||
|
Caddy still targeted the fixed address during that restart, so the Observatory
|
||||||
|
agent observed a small number of transient HTTP `502` responses. Its durable
|
||||||
|
spool retried successfully and no accepted telemetry was lost, but the routed
|
||||||
|
origin was not continuously available.
|
||||||
|
|
||||||
|
The corrected activation contract treats the candidate as a traffic handoff,
|
||||||
|
not merely a preflight process. Tend validates and routes the imported Caddy
|
||||||
|
handler to the candidate before changing the release pointer. It restarts and
|
||||||
|
probes the fixed-address unit without public traffic, routes back only after
|
||||||
|
that unit passes, and keeps the candidate healthy throughout the bounded
|
||||||
|
activation window. Any failure restores the former pointer and exact handler
|
||||||
|
bytes. If restoration itself cannot be completed, Tend leaves the proven
|
||||||
|
candidate routed and running for explicit operator recovery instead of causing
|
||||||
|
a known outage.
|
||||||
|
|
||||||
|
Regression tests cover successful handoff, restart failure, public-origin
|
||||||
|
failure during the activation window, pointer and handler restoration, Caddy
|
||||||
|
validation/reload boundaries, and candidate cleanup. Final production evidence
|
||||||
|
must still repeat deploy, rollback, and reactivation with the exact release
|
||||||
|
binary before this finding is treated as released.
|
||||||
|
|
||||||
|
## F-10: archive modes under a hardened umask
|
||||||
|
|
||||||
|
### Observation
|
||||||
|
|
||||||
|
The first two August 18 maintenance candidates stopped before activation. The
|
||||||
|
release archives correctly recorded executable mode `0755`, but extraction by
|
||||||
|
root under umask `0077` left the installed application binary mode `0700`.
|
||||||
|
The unprivileged transient candidate could not execute it. Tend emitted failed
|
||||||
|
candidate evidence, retained the prior release and route, and did not expose
|
||||||
|
the failed binary to traffic.
|
||||||
|
|
||||||
|
Extraction now reapplies the already validated archive mode after file content
|
||||||
|
is closed. A Linux regression test sets umask `0077`, extracts the release, and
|
||||||
|
requires the installed binary to remain executable by the service identity.
|
||||||
|
The successful maintenance campaign used that corrected Tend binary.
|
||||||
|
|
||||||
|
## Prioritization
|
||||||
|
|
||||||
|
The recommended implementation order is:
|
||||||
|
|
||||||
|
1. final production proof of the singleton traffic handoff;
|
||||||
|
2. a checksum-verified resolver and network-disabled package contract;
|
||||||
|
3. an early linked-worktree diagnostic and a provenance-preserving support
|
||||||
|
decision;
|
||||||
|
4. `tend inspect` and a standardized CI artifact/approval contract;
|
||||||
|
5. exact release plus configuration identity in status and state;
|
||||||
|
6. a bounded application preflight contract;
|
||||||
|
7. restricted transfer and receive with host policy;
|
||||||
|
8. explicit adoption for existing services;
|
||||||
|
9. managed configuration only after its restoration and failure-injection
|
||||||
|
model is as strong as binary activation.
|
||||||
|
|
||||||
|
Friction entries should be updated with the implementing version, tests, and
|
||||||
|
dogfood evidence when resolved. Resolved entries remain in this ledger so the
|
||||||
|
reason for the security boundary is not lost.
|
||||||
@@ -12,3 +12,8 @@ material markers. The resulting directory receives a new public root commit.
|
|||||||
Public Gitea is canonical for issues, contributions, and releases. GitHub is a
|
Public Gitea is canonical for issues, contributions, and releases. GitHub is a
|
||||||
read-only discovery copy of the same public tree. Tags belong only to canonical
|
read-only discovery copy of the same public tree. Tags belong only to canonical
|
||||||
Gitea.
|
Gitea.
|
||||||
|
|
||||||
|
The snapshot includes the program, security and architecture documentation,
|
||||||
|
copyable examples, release configuration, and local verification scripts.
|
||||||
|
Private workflows, runner configuration, repository credentials, and raw
|
||||||
|
operational evidence remain outside the public root.
|
||||||
|
|||||||
@@ -0,0 +1,35 @@
|
|||||||
|
# Schema 1 to schema 2
|
||||||
|
|
||||||
|
Schema 2 is intentionally not loaded as schema 1. Keep the installed v0.1 Tend
|
||||||
|
binary available until the first schema-2 activation and rollback have both
|
||||||
|
been exercised.
|
||||||
|
|
||||||
|
1. Move each configuration to `/etc/tend/services/<service>.json`.
|
||||||
|
2. Set `schema_version` to `2`.
|
||||||
|
3. Add `service.environment_file` below `/etc/tend/environment/`.
|
||||||
|
4. Create that file as a regular root-owned file with mode `0600`. Move secret
|
||||||
|
values out of JSON. Do not put the environment file in Git. For a singleton,
|
||||||
|
keep its `listen_env` key out of this shared file: set the live address in
|
||||||
|
the installed unit and let Tend supply only the candidate address.
|
||||||
|
5. Set every service's `deployment.lock_file` to
|
||||||
|
`/run/lock/tend-deploy.lock`.
|
||||||
|
6. Add one or more query-free HTTPS `deployment.public_smoke` checks.
|
||||||
|
7. Add a per-service `deployment.event_log` below its release root and a
|
||||||
|
bounded `deployment.activation_window_seconds` value. Keep the log
|
||||||
|
root-owned and grant collectors read access explicitly.
|
||||||
|
8. For singleton services, add the full Caddy configuration, imported handler,
|
||||||
|
and one-upstream handler-template paths. The template must be reviewed and
|
||||||
|
contain exactly one `{{UPSTREAM}}` marker so Tend can keep traffic on the
|
||||||
|
candidate while the fixed-address unit restarts.
|
||||||
|
9. Update installed systemd units to read the same environment file as the
|
||||||
|
transient candidate.
|
||||||
|
10. Install a root-owned `0600` receive policy mapping each service name to its
|
||||||
|
exact configuration and artifact-size ceiling.
|
||||||
|
11. Run `tend check-server` as root before accepting a transfer.
|
||||||
|
12. Validate, activate, rollback, and reactivate one service at a time. Confirm
|
||||||
|
unrelated services never restart.
|
||||||
|
|
||||||
|
Tend does not discover `.env`, infer old values, rewrite a production file, or
|
||||||
|
silently migrate state. For local development, copy a committed `.env.example`
|
||||||
|
to an ignored `.env.local`, restrict its mode, and load it with the application's
|
||||||
|
own tooling. Tend never loads local dotenv files implicitly.
|
||||||
+30
-14
@@ -7,33 +7,49 @@
|
|||||||
- Release roots, state, pointers, and Caddy files reject symlink substitution
|
- Release roots, state, pointers, and Caddy files reject symlink substitution
|
||||||
at their checked boundaries.
|
at their checked boundaries.
|
||||||
- Configuration is strict JSON and is never interpolated into a shell command.
|
- Configuration is strict JSON and is never interpolated into a shell command.
|
||||||
- Candidate health is established before traffic or the singleton current
|
- Candidate health is established before traffic or any current release
|
||||||
pointer changes. New deployments also satisfy configured content smoke
|
pointer changes. New deployments also satisfy configured content smoke
|
||||||
checks; rollback uses health and readiness because future-release content
|
checks; rollback uses health and readiness because future-release content
|
||||||
markers are not valid requirements for an older retained release.
|
markers are not valid requirements for an older retained release.
|
||||||
- Caddy configuration validates before reload.
|
- Caddy configuration validates before reload.
|
||||||
|
- Canonical routed origins and the previous blue/green slot or singleton
|
||||||
|
handoff candidate remain under probe for the configured activation window; a
|
||||||
|
failure restores the old handler and release pointer.
|
||||||
|
- Desired, candidate, active, previous, and failed-attempt identities remain
|
||||||
|
distinct in state. Bounded deployment events contain no arbitrary command
|
||||||
|
output or environment values and cannot block deployment.
|
||||||
- An activation failure restores the previously observed state.
|
- An activation failure restores the previously observed state.
|
||||||
- Active and previous releases survive pruning.
|
- Active and previous releases survive pruning.
|
||||||
|
- The restricted receiver accepts one versioned bounded stream, one allowlisted
|
||||||
|
service name, and one explicitly approved digest. It accepts no remote path,
|
||||||
|
URL, environment value, or shell expression.
|
||||||
|
- OpenSSH host keys are pinned; forwarding, PTYs, proxy commands, local commands,
|
||||||
|
and user SSH configuration are disabled by the client invocation.
|
||||||
|
- Production secret values are absent from configuration, process arguments,
|
||||||
|
artifacts, manifests, reports, deployment state, and Tend logs.
|
||||||
|
- One host-wide lock serializes shared Caddy activation without stopping or
|
||||||
|
reconfiguring unrelated services.
|
||||||
|
|
||||||
## Trusted inputs
|
## Trusted inputs
|
||||||
|
|
||||||
Reviewed source, the Go toolchain, root-owned `tend.json`, systemd unit files,
|
Reviewed source, the pinned Go toolchain, root-owned server policy and service
|
||||||
the Caddy handler template, the operator, and the target host's root account are
|
configuration, systemd unit files, the Caddy handler template, the operator,
|
||||||
trusted. Packaging a hostile repository can execute hostile Go compiler hooks
|
and the target host's root account are trusted. Packaging a hostile repository
|
||||||
or consume resources; Tend is not a source-code sandbox.
|
can execute hostile Go compiler hooks or consume resources; Tend is not a
|
||||||
|
source-code sandbox.
|
||||||
|
|
||||||
## Adversarial inputs
|
## Adversarial inputs
|
||||||
|
|
||||||
Artifact paths and bytes, archive metadata, stale or malformed state, HTTP
|
Artifact paths and bytes, protocol frames, requested service names, archive
|
||||||
responses, subprocess failures, and filesystem objects at managed paths are
|
metadata, stale or malformed state, HTTP responses, subprocess failures, and
|
||||||
validated and fail closed. Process output is bounded.
|
filesystem objects at managed paths are validated and fail closed. Process
|
||||||
|
output and receive fields are bounded.
|
||||||
|
|
||||||
## Non-goals and preview limits
|
## Non-goals and preview limits
|
||||||
|
|
||||||
Tend does not defend against a concurrently malicious root user, a compromised
|
Tend does not defend against a concurrently malicious root user, a compromised
|
||||||
kernel/toolchain/systemd/Caddy installation, denial of service by the trusted
|
kernel/toolchain/systemd/Caddy/OpenSSH installation, denial of service by the
|
||||||
application, or secrets already exposed through service configuration. v0.1
|
trusted application, or secrets an application itself exposes. It does not
|
||||||
does not manage data migrations, databases, containers, Kubernetes, or
|
manage data migrations, databases, containers, Kubernetes, or application-specific
|
||||||
application-specific activation. Transfer remains an application-local,
|
activation. EQL therefore remains outside the generic deployment adapter until
|
||||||
reviewed step until a transport can be added without weakening host-key and
|
its SQLite/catalog publication checkpoints can be modeled explicitly.
|
||||||
credential boundaries.
|
|
||||||
|
|||||||
@@ -0,0 +1,70 @@
|
|||||||
|
# Two-service build, approval, and deployment
|
||||||
|
|
||||||
|
This walkthrough assumes one small Linux host running systemd and Caddy, two Go
|
||||||
|
services, and a trusted Gitea build runner. The host has the same root-owned Tend
|
||||||
|
binary at `/usr/local/bin/tend`; it does not need Git or Go.
|
||||||
|
|
||||||
|
## Prepare the host once
|
||||||
|
|
||||||
|
1. Create `/etc/tend/services`, `/etc/tend/environment`, and
|
||||||
|
`/var/lib/tend/incoming`. The incoming and environment directories are
|
||||||
|
root-owned mode `0700`.
|
||||||
|
2. Install one schema-2 file per service, one per-service deployment-event log,
|
||||||
|
and one root-owned mode-`0600`
|
||||||
|
environment file per service. A singleton's shared environment file must
|
||||||
|
not define its configured listen key; its installed unit owns the live
|
||||||
|
address and Tend overrides only the transient candidate. Give each
|
||||||
|
singleton an imported Caddy handler and a root-owned handler template with
|
||||||
|
exactly one `{{UPSTREAM}}` marker; this is the bounded traffic handoff while
|
||||||
|
its fixed-address unit restarts.
|
||||||
|
3. Install the receive policy, forced `authorized_keys` entry, and exact sudoers
|
||||||
|
rule from `examples/server/` after replacing every placeholder. The sudoers
|
||||||
|
fragment preserves only `SSH_ORIGINAL_COMMAND`; the root receiver requires
|
||||||
|
its exact protocol value and refuses every other requested command.
|
||||||
|
4. Pin the server host key in a dedicated client file. Do not accept a new key
|
||||||
|
interactively during deployment.
|
||||||
|
5. Run `sudo tend check-server` and inspect the allowlisted service names.
|
||||||
|
|
||||||
|
## Build and approve
|
||||||
|
|
||||||
|
Trusted CI checks a clean pushed commit, uses a pinned Go toolchain, packages
|
||||||
|
twice, and requires byte-identical archives. It publishes the archive,
|
||||||
|
`RELEASE.json`, `BUILDINFO.json`, SPDX SBOM, and SHA-256 evidence.
|
||||||
|
|
||||||
|
The maintainer reads the candidate report and copies the exact approved digest
|
||||||
|
into the deployment command. Tend refuses a digest that is merely inferred from
|
||||||
|
the local file or differs from the produced value.
|
||||||
|
|
||||||
|
```text
|
||||||
|
tend push --target tend-deploy@server.example \
|
||||||
|
--known-hosts /secure/tend_known_hosts \
|
||||||
|
--identity /secure/tend_deploy_ed25519 \
|
||||||
|
--service example-site \
|
||||||
|
--artifact /approved/example-site.tar.gz \
|
||||||
|
--sha256 <produced> --approve-sha256 <reviewed> --activate
|
||||||
|
```
|
||||||
|
|
||||||
|
Repeat independently with `--service docs-site`. Both builds and transfers can
|
||||||
|
run concurrently. The host-wide lock serializes only activation because both
|
||||||
|
services share Caddy. Tend does not stop the other application.
|
||||||
|
|
||||||
|
## Failure and recovery exercises
|
||||||
|
|
||||||
|
- Change a candidate marker: activation must fail, preserve the active release,
|
||||||
|
and record the failed attempt without calling the candidate active.
|
||||||
|
- Make a Caddy template invalid: validation must fail and restore prior bytes.
|
||||||
|
- Make the public marker fail after an initially successful request: the
|
||||||
|
activation window must catch the transient routed failure and restore the
|
||||||
|
former slot/pointer.
|
||||||
|
- For a singleton, verify repeated canonical-origin requests remain successful
|
||||||
|
while Tend routes to the candidate, restarts the fixed-address unit, and
|
||||||
|
returns traffic to it. Inject failure at both Caddy reloads and require the
|
||||||
|
prior handler and pointer to be restored.
|
||||||
|
- Stop the previous blue/green slot during the activation window: Tend must
|
||||||
|
restore the old Caddy handler instead of accepting reduced continuity.
|
||||||
|
- Run `tend rollback --activate` for one service and verify the other service's
|
||||||
|
units, pointers, and public origin did not change.
|
||||||
|
- Interrupt a transfer: no release becomes active and the incomplete incoming
|
||||||
|
file is removed when the receiver exits.
|
||||||
|
|
||||||
|
After the soak, prune per service. Active and previous releases remain protected.
|
||||||
+28
-6
@@ -3,17 +3,39 @@
|
|||||||
This subtree is licensed 0BSD so an operator can copy and adapt it without
|
This subtree is licensed 0BSD so an operator can copy and adapt it without
|
||||||
bringing the Tend program's AGPL license into an application configuration.
|
bringing the Tend program's AGPL license into an application configuration.
|
||||||
|
|
||||||
The blue/green example expects separately reviewed environment files:
|
The blue/green example expects one separately reviewed environment file that
|
||||||
|
the two installed slots and transient validation use consistently:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
# /etc/example-site/blue.env
|
# /etc/tend/environment/example-site.env
|
||||||
EXAMPLE_LISTEN=127.0.0.1:8090
|
APP_SECRET=replace-on-server
|
||||||
|
|
||||||
# /etc/example-site/green.env
|
|
||||||
EXAMPLE_LISTEN=127.0.0.1:8091
|
|
||||||
```
|
```
|
||||||
|
|
||||||
|
The blue/green systemd slot units then read the nonsecret listen address from
|
||||||
|
`/etc/tend/slots/example-site-blue.env` or `-green.env`; Tend overrides only
|
||||||
|
the isolated candidate address. Secret values never enter `tend.json`.
|
||||||
|
|
||||||
|
The singleton example follows the same split: its shared root-only environment
|
||||||
|
file omits the configured listen key, the installed unit owns the live address,
|
||||||
|
and Tend supplies only the transient candidate address. This prevents a shared
|
||||||
|
environment file from overriding the isolated candidate port. Its imported
|
||||||
|
Caddy handler is also managed from one reviewed template. Tend temporarily
|
||||||
|
routes the canonical origin to the proven candidate while the fixed-address
|
||||||
|
unit restarts, then returns traffic to that unit only after it passes its local
|
||||||
|
checks.
|
||||||
|
|
||||||
Production configuration belongs outside the source checkout, owned by root,
|
Production configuration belongs outside the source checkout, owned by root,
|
||||||
and not group- or world-writable. The Caddy handler template is an entire
|
and not group- or world-writable. The Caddy handler template is an entire
|
||||||
imported handler fragment; the enclosing site, matchers, and routing precedence
|
imported handler fragment; the enclosing site, matchers, and routing precedence
|
||||||
remain operator-owned.
|
remain operator-owned.
|
||||||
|
|
||||||
|
`event_log` is a per-service, root-owned JSONL evidence stream below that
|
||||||
|
service's release root. Grant an Observatory agent read access explicitly; do
|
||||||
|
not make the release root broadly readable. `activation_window_seconds` keeps
|
||||||
|
canonical routed probes active after Caddy reload and keeps the previous
|
||||||
|
blue/green slot—or the singleton handoff candidate—under health/readiness
|
||||||
|
observation until the activation is recorded.
|
||||||
|
|
||||||
|
`server/` demonstrates the schema-2 receive policy, forced OpenSSH command,
|
||||||
|
restricted sudo entry, two independent service configurations, and secret-file
|
||||||
|
placement. The values are placeholders, not an installation script.
|
||||||
|
|||||||
@@ -6,7 +6,8 @@ After=network.target
|
|||||||
Type=simple
|
Type=simple
|
||||||
DynamicUser=yes
|
DynamicUser=yes
|
||||||
ExecStart=/opt/example-site/slots/%i/example-site
|
ExecStart=/opt/example-site/slots/%i/example-site
|
||||||
EnvironmentFile=/etc/example-site/%i.env
|
EnvironmentFile=/etc/tend/environment/example-site.env
|
||||||
|
EnvironmentFile=/etc/tend/slots/example-site-%i.env
|
||||||
NoNewPrivileges=yes
|
NoNewPrivileges=yes
|
||||||
PrivateTmp=yes
|
PrivateTmp=yes
|
||||||
ProtectSystem=strict
|
ProtectSystem=strict
|
||||||
|
|||||||
@@ -1,16 +1,19 @@
|
|||||||
{
|
{
|
||||||
"schema_version": 1,
|
"schema_version": 2,
|
||||||
"service": { "name": "example-site", "allowed_host": "example.test" },
|
"service": { "name": "example-site", "allowed_host": "example.test", "environment_file": "/etc/tend/environment/example-site.env" },
|
||||||
"build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" },
|
"build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" },
|
||||||
"deployment": {
|
"deployment": {
|
||||||
"strategy": "blue_green",
|
"strategy": "blue_green",
|
||||||
"root": "/opt/example-site",
|
"root": "/opt/example-site",
|
||||||
"lock_file": "/run/lock/gamertan-deploy.lock",
|
"lock_file": "/run/lock/tend-deploy.lock",
|
||||||
"state_file": "/opt/example-site/tend-state.json",
|
"state_file": "/opt/example-site/tend-state.json",
|
||||||
|
"event_log": "/opt/example-site/deployment-events.jsonl",
|
||||||
"health_path": "/healthz",
|
"health_path": "/healthz",
|
||||||
"readiness_path": "/readyz",
|
"readiness_path": "/readyz",
|
||||||
"candidate_timeout_seconds": 30,
|
"candidate_timeout_seconds": 30,
|
||||||
|
"activation_window_seconds": 10,
|
||||||
"smoke": [{ "path": "/", "contains": "Example site" }],
|
"smoke": [{ "path": "/", "contains": "Example site" }],
|
||||||
|
"public_smoke": [{ "url": "https://example.test/", "contains": "Example site" }],
|
||||||
"blue_green": {
|
"blue_green": {
|
||||||
"caddy_config": "/etc/caddy/Caddyfile",
|
"caddy_config": "/etc/caddy/Caddyfile",
|
||||||
"caddy_handler": "/etc/caddy/example-site-handler.caddy",
|
"caddy_handler": "/etc/caddy/example-site-handler.caddy",
|
||||||
|
|||||||
@@ -0,0 +1,3 @@
|
|||||||
|
# SPDX-License-Identifier: 0BSD
|
||||||
|
APP_MODE=development
|
||||||
|
APP_SECRET=replace-with-a-local-random-value
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# SPDX-License-Identifier: 0BSD
|
||||||
|
restrict,command="sudo -n /usr/local/bin/tend receive --policy /etc/tend/receive-policy.json" ssh-ed25519 REPLACE_WITH_DEPLOY_KEY tend-deploy
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# Managed by Tend. The enclosing site and route matchers remain operator-owned.
|
||||||
|
reverse_proxy {{UPSTREAM}}
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# Managed by Tend. The enclosing site and route matchers remain operator-owned.
|
||||||
|
reverse_proxy {{UPSTREAM}}
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
# SPDX-License-Identifier: 0BSD
|
||||||
|
DOCS_LISTEN=127.0.0.1:8102
|
||||||
|
APP_SECRET=replace-on-server
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# SPDX-License-Identifier: 0BSD
|
||||||
|
APP_SECRET=replace-on-server
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
# SPDX-License-Identifier: 0BSD
|
||||||
|
[Unit]
|
||||||
|
Description=Example singleton site
|
||||||
|
After=network.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
DynamicUser=yes
|
||||||
|
ExecStart=/opt/example-site/current/example-site
|
||||||
|
EnvironmentFile=/etc/tend/environment/example-site.env
|
||||||
|
Environment=EXAMPLE_LISTEN=127.0.0.1:8092
|
||||||
|
NoNewPrivileges=yes
|
||||||
|
PrivateTmp=yes
|
||||||
|
ProtectSystem=strict
|
||||||
|
ProtectHome=yes
|
||||||
|
Restart=on-failure
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target
|
||||||
@@ -0,0 +1,16 @@
|
|||||||
|
{
|
||||||
|
"schema_version": 1,
|
||||||
|
"config_root": "/etc/tend/services",
|
||||||
|
"incoming_root": "/var/lib/tend/incoming",
|
||||||
|
"shared_lock_file": "/run/lock/tend-deploy.lock",
|
||||||
|
"services": {
|
||||||
|
"docs-site": {
|
||||||
|
"config": "/etc/tend/services/docs-site.json",
|
||||||
|
"max_artifact_bytes": 134217728
|
||||||
|
},
|
||||||
|
"example-site": {
|
||||||
|
"config": "/etc/tend/services/example-site.json",
|
||||||
|
"max_artifact_bytes": 134217728
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
{
|
||||||
|
"schema_version": 2,
|
||||||
|
"service": { "name": "docs-site", "allowed_host": "docs.example.test", "environment_file": "/etc/tend/environment/docs-site.env" },
|
||||||
|
"build": { "package": "./cmd/docs", "binary": "docs-site", "branch": "main" },
|
||||||
|
"deployment": {
|
||||||
|
"strategy": "singleton_candidate",
|
||||||
|
"root": "/opt/docs-site",
|
||||||
|
"lock_file": "/run/lock/tend-deploy.lock",
|
||||||
|
"state_file": "/opt/docs-site/tend-state.json",
|
||||||
|
"event_log": "/opt/docs-site/deployment-events.jsonl",
|
||||||
|
"health_path": "/healthz",
|
||||||
|
"readiness_path": "/readyz",
|
||||||
|
"candidate_timeout_seconds": 30,
|
||||||
|
"activation_window_seconds": 10,
|
||||||
|
"smoke": [{ "path": "/", "contains": "Documentation" }],
|
||||||
|
"public_smoke": [{ "url": "https://docs.example.test/", "contains": "Documentation" }],
|
||||||
|
"singleton": {
|
||||||
|
"unit": "docs-site.service",
|
||||||
|
"address": "127.0.0.1:8102",
|
||||||
|
"candidate_address": "127.0.0.1:18102",
|
||||||
|
"listen_env": "DOCS_LISTEN",
|
||||||
|
"current_link": "/opt/docs-site/current",
|
||||||
|
"previous_link": "/opt/docs-site/previous",
|
||||||
|
"caddy_config": "/etc/caddy/Caddyfile",
|
||||||
|
"caddy_handler": "/etc/caddy/docs-site-handler.caddy",
|
||||||
|
"caddy_handler_template": "/etc/tend/caddy/docs-site.template"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
{
|
||||||
|
"schema_version": 2,
|
||||||
|
"service": { "name": "example-site", "allowed_host": "example.test", "environment_file": "/etc/tend/environment/example-site.env" },
|
||||||
|
"build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" },
|
||||||
|
"deployment": {
|
||||||
|
"strategy": "singleton_candidate",
|
||||||
|
"root": "/opt/example-site",
|
||||||
|
"lock_file": "/run/lock/tend-deploy.lock",
|
||||||
|
"state_file": "/opt/example-site/tend-state.json",
|
||||||
|
"event_log": "/opt/example-site/deployment-events.jsonl",
|
||||||
|
"health_path": "/healthz",
|
||||||
|
"readiness_path": "/readyz",
|
||||||
|
"candidate_timeout_seconds": 30,
|
||||||
|
"activation_window_seconds": 10,
|
||||||
|
"smoke": [{ "path": "/", "contains": "Example site" }],
|
||||||
|
"public_smoke": [{ "url": "https://example.test/", "contains": "Example site" }],
|
||||||
|
"singleton": {
|
||||||
|
"unit": "example-site.service",
|
||||||
|
"address": "127.0.0.1:8092",
|
||||||
|
"candidate_address": "127.0.0.1:18092",
|
||||||
|
"listen_env": "EXAMPLE_LISTEN",
|
||||||
|
"current_link": "/opt/example-site/current",
|
||||||
|
"previous_link": "/opt/example-site/previous",
|
||||||
|
"caddy_config": "/etc/caddy/Caddyfile",
|
||||||
|
"caddy_handler": "/etc/caddy/example-site-handler.caddy",
|
||||||
|
"caddy_handler_template": "/etc/tend/caddy/example-site.template"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# SPDX-License-Identifier: 0BSD
|
||||||
|
EXAMPLE_LISTEN=127.0.0.1:8090
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# SPDX-License-Identifier: 0BSD
|
||||||
|
EXAMPLE_LISTEN=127.0.0.1:8091
|
||||||
@@ -0,0 +1,3 @@
|
|||||||
|
# SPDX-License-Identifier: 0BSD
|
||||||
|
Defaults:tend-deploy env_keep += "SSH_ORIGINAL_COMMAND"
|
||||||
|
tend-deploy ALL=(root) NOPASSWD: /usr/local/bin/tend receive --policy /etc/tend/receive-policy.json
|
||||||
@@ -0,0 +1,2 @@
|
|||||||
|
# Managed by Tend. The enclosing site and route matchers remain operator-owned.
|
||||||
|
reverse_proxy {{UPSTREAM}}
|
||||||
@@ -1,24 +1,29 @@
|
|||||||
{
|
{
|
||||||
"schema_version": 1,
|
"schema_version": 2,
|
||||||
"service": { "name": "example-site", "allowed_host": "example.test" },
|
"service": { "name": "example-site", "allowed_host": "example.test", "environment_file": "/etc/tend/environment/example-site.env" },
|
||||||
"build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" },
|
"build": { "package": "./cmd/site", "binary": "example-site", "branch": "main" },
|
||||||
"deployment": {
|
"deployment": {
|
||||||
"strategy": "singleton_candidate",
|
"strategy": "singleton_candidate",
|
||||||
"root": "/opt/example-site",
|
"root": "/opt/example-site",
|
||||||
"lock_file": "/run/lock/gamertan-deploy.lock",
|
"lock_file": "/run/lock/tend-deploy.lock",
|
||||||
"state_file": "/opt/example-site/tend-state.json",
|
"state_file": "/opt/example-site/tend-state.json",
|
||||||
|
"event_log": "/opt/example-site/deployment-events.jsonl",
|
||||||
"health_path": "/healthz",
|
"health_path": "/healthz",
|
||||||
"readiness_path": "/readyz",
|
"readiness_path": "/readyz",
|
||||||
"candidate_timeout_seconds": 30,
|
"candidate_timeout_seconds": 30,
|
||||||
|
"activation_window_seconds": 10,
|
||||||
"smoke": [{ "path": "/", "contains": "Example site" }],
|
"smoke": [{ "path": "/", "contains": "Example site" }],
|
||||||
|
"public_smoke": [{ "url": "https://example.test/", "contains": "Example site" }],
|
||||||
"singleton": {
|
"singleton": {
|
||||||
"unit": "example-site.service",
|
"unit": "example-site.service",
|
||||||
"address": "127.0.0.1:8092",
|
"address": "127.0.0.1:8092",
|
||||||
"candidate_address": "127.0.0.1:18092",
|
"candidate_address": "127.0.0.1:18092",
|
||||||
"listen_env": "EXAMPLE_LISTEN",
|
"listen_env": "EXAMPLE_LISTEN",
|
||||||
"environment": {},
|
|
||||||
"current_link": "/opt/example-site/current",
|
"current_link": "/opt/example-site/current",
|
||||||
"previous_link": "/opt/example-site/previous"
|
"previous_link": "/opt/example-site/previous",
|
||||||
|
"caddy_config": "/etc/caddy/Caddyfile",
|
||||||
|
"caddy_handler": "/etc/caddy/example-site-handler.caddy",
|
||||||
|
"caddy_handler_template": "/etc/tend/caddy/example-site.template"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,13 +9,17 @@ import (
|
|||||||
"fmt"
|
"fmt"
|
||||||
"io"
|
"io"
|
||||||
"net/netip"
|
"net/netip"
|
||||||
|
"net/url"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"regexp"
|
"regexp"
|
||||||
"strings"
|
"strings"
|
||||||
)
|
)
|
||||||
|
|
||||||
const SchemaVersion = 1
|
const (
|
||||||
|
SchemaVersion = 2
|
||||||
|
SharedLockFile = "/run/lock/tend-deploy.lock"
|
||||||
|
)
|
||||||
|
|
||||||
var (
|
var (
|
||||||
namePattern = regexp.MustCompile(`^[a-z][a-z0-9-]{1,62}$`)
|
namePattern = regexp.MustCompile(`^[a-z][a-z0-9-]{1,62}$`)
|
||||||
@@ -35,6 +39,7 @@ type Config struct {
|
|||||||
type Service struct {
|
type Service struct {
|
||||||
Name string `json:"name"`
|
Name string `json:"name"`
|
||||||
AllowedHost string `json:"allowed_host"`
|
AllowedHost string `json:"allowed_host"`
|
||||||
|
EnvironmentFile string `json:"environment_file"`
|
||||||
}
|
}
|
||||||
|
|
||||||
type Build struct {
|
type Build struct {
|
||||||
@@ -51,10 +56,13 @@ type Deployment struct {
|
|||||||
Root string `json:"root"`
|
Root string `json:"root"`
|
||||||
LockFile string `json:"lock_file"`
|
LockFile string `json:"lock_file"`
|
||||||
StateFile string `json:"state_file"`
|
StateFile string `json:"state_file"`
|
||||||
|
EventLog string `json:"event_log"`
|
||||||
HealthPath string `json:"health_path"`
|
HealthPath string `json:"health_path"`
|
||||||
ReadinessPath string `json:"readiness_path"`
|
ReadinessPath string `json:"readiness_path"`
|
||||||
CandidateTimeoutSecs int `json:"candidate_timeout_seconds"`
|
CandidateTimeoutSecs int `json:"candidate_timeout_seconds"`
|
||||||
|
ActivationWindowSecs int `json:"activation_window_seconds"`
|
||||||
Smoke []Smoke `json:"smoke"`
|
Smoke []Smoke `json:"smoke"`
|
||||||
|
PublicSmoke []PublicSmoke `json:"public_smoke"`
|
||||||
BlueGreen *BlueGreen `json:"blue_green,omitempty"`
|
BlueGreen *BlueGreen `json:"blue_green,omitempty"`
|
||||||
Singleton *Singleton `json:"singleton,omitempty"`
|
Singleton *Singleton `json:"singleton,omitempty"`
|
||||||
}
|
}
|
||||||
@@ -64,6 +72,11 @@ type Smoke struct {
|
|||||||
Contains string `json:"contains"`
|
Contains string `json:"contains"`
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type PublicSmoke struct {
|
||||||
|
URL string `json:"url"`
|
||||||
|
Contains string `json:"contains"`
|
||||||
|
}
|
||||||
|
|
||||||
type BlueGreen struct {
|
type BlueGreen struct {
|
||||||
CaddyConfig string `json:"caddy_config"`
|
CaddyConfig string `json:"caddy_config"`
|
||||||
CaddyHandler string `json:"caddy_handler"`
|
CaddyHandler string `json:"caddy_handler"`
|
||||||
@@ -84,9 +97,11 @@ type Singleton struct {
|
|||||||
Address string `json:"address"`
|
Address string `json:"address"`
|
||||||
CandidateAddress string `json:"candidate_address"`
|
CandidateAddress string `json:"candidate_address"`
|
||||||
ListenEnv string `json:"listen_env"`
|
ListenEnv string `json:"listen_env"`
|
||||||
Environment map[string]string `json:"environment,omitempty"`
|
|
||||||
CurrentLink string `json:"current_link"`
|
CurrentLink string `json:"current_link"`
|
||||||
PreviousLink string `json:"previous_link"`
|
PreviousLink string `json:"previous_link"`
|
||||||
|
CaddyConfig string `json:"caddy_config"`
|
||||||
|
CaddyHandler string `json:"caddy_handler"`
|
||||||
|
CaddyHandlerTemplate string `json:"caddy_handler_template"`
|
||||||
}
|
}
|
||||||
|
|
||||||
func Load(path string) (Config, error) {
|
func Load(path string) (Config, error) {
|
||||||
@@ -140,6 +155,12 @@ func (c Config) Validate() error {
|
|||||||
if c.Service.AllowedHost == "" || strings.ContainsAny(c.Service.AllowedHost, "/\\\x00\r\n\t ") {
|
if c.Service.AllowedHost == "" || strings.ContainsAny(c.Service.AllowedHost, "/\\\x00\r\n\t ") {
|
||||||
return errors.New("service.allowed_host is invalid")
|
return errors.New("service.allowed_host is invalid")
|
||||||
}
|
}
|
||||||
|
if err := safeAbsolute("service.environment_file", c.Service.EnvironmentFile); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !within("/etc/tend/environment", c.Service.EnvironmentFile) {
|
||||||
|
return errors.New("service.environment_file must be below /etc/tend/environment")
|
||||||
|
}
|
||||||
if !packagePattern.MatchString(c.Build.Package) || strings.Contains(c.Build.Package, "..") {
|
if !packagePattern.MatchString(c.Build.Package) || strings.Contains(c.Build.Package, "..") {
|
||||||
return errors.New("build.package must be a local package without traversal")
|
return errors.New("build.package must be a local package without traversal")
|
||||||
}
|
}
|
||||||
@@ -171,12 +192,21 @@ func (c Config) Validate() error {
|
|||||||
if filepath.Clean(d.StateFile) == filepath.Clean(d.Root) || !within(d.Root, d.StateFile) {
|
if filepath.Clean(d.StateFile) == filepath.Clean(d.Root) || !within(d.Root, d.StateFile) {
|
||||||
return errors.New("deployment.state_file must be below deployment.root")
|
return errors.New("deployment.state_file must be below deployment.root")
|
||||||
}
|
}
|
||||||
|
if err := safeAbsolute("deployment.event_log", d.EventLog); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if filepath.Clean(d.EventLog) == filepath.Clean(d.Root) || !within(d.Root, d.EventLog) || filepath.Clean(d.EventLog) == filepath.Clean(d.StateFile) {
|
||||||
|
return errors.New("deployment.event_log must be a distinct file below deployment.root")
|
||||||
|
}
|
||||||
if !safeHTTPPath(d.HealthPath) || !safeHTTPPath(d.ReadinessPath) {
|
if !safeHTTPPath(d.HealthPath) || !safeHTTPPath(d.ReadinessPath) {
|
||||||
return errors.New("health and readiness paths must be absolute HTTP paths")
|
return errors.New("health and readiness paths must be absolute HTTP paths")
|
||||||
}
|
}
|
||||||
if d.CandidateTimeoutSecs < 2 || d.CandidateTimeoutSecs > 300 {
|
if d.CandidateTimeoutSecs < 2 || d.CandidateTimeoutSecs > 300 {
|
||||||
return errors.New("candidate_timeout_seconds must be between 2 and 300")
|
return errors.New("candidate_timeout_seconds must be between 2 and 300")
|
||||||
}
|
}
|
||||||
|
if d.ActivationWindowSecs < 1 || d.ActivationWindowSecs > 120 {
|
||||||
|
return errors.New("activation_window_seconds must be between 1 and 120")
|
||||||
|
}
|
||||||
if len(d.Smoke) == 0 || len(d.Smoke) > 32 {
|
if len(d.Smoke) == 0 || len(d.Smoke) > 32 {
|
||||||
return errors.New("deployment.smoke must contain 1 to 32 checks")
|
return errors.New("deployment.smoke must contain 1 to 32 checks")
|
||||||
}
|
}
|
||||||
@@ -185,6 +215,15 @@ func (c Config) Validate() error {
|
|||||||
return fmt.Errorf("deployment.smoke[%d] is invalid", i)
|
return fmt.Errorf("deployment.smoke[%d] is invalid", i)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if len(d.PublicSmoke) == 0 || len(d.PublicSmoke) > 16 {
|
||||||
|
return errors.New("deployment.public_smoke must contain 1 to 16 checks")
|
||||||
|
}
|
||||||
|
for i, smoke := range d.PublicSmoke {
|
||||||
|
parsed, err := url.Parse(smoke.URL)
|
||||||
|
if err != nil || parsed.Scheme != "https" || parsed.Host == "" || parsed.User != nil || parsed.Fragment != "" || parsed.RawQuery != "" || parsed.Opaque != "" || smoke.Contains == "" || len(smoke.Contains) > 4096 || strings.ContainsRune(smoke.Contains, '\x00') {
|
||||||
|
return fmt.Errorf("deployment.public_smoke[%d] is invalid", i)
|
||||||
|
}
|
||||||
|
}
|
||||||
switch d.Strategy {
|
switch d.Strategy {
|
||||||
case "blue_green":
|
case "blue_green":
|
||||||
if d.BlueGreen == nil || d.Singleton != nil {
|
if d.BlueGreen == nil || d.Singleton != nil {
|
||||||
@@ -273,11 +312,14 @@ func validateSingleton(root string, s Singleton) error {
|
|||||||
if s.CurrentLink == s.PreviousLink {
|
if s.CurrentLink == s.PreviousLink {
|
||||||
return errors.New("current and previous links must differ")
|
return errors.New("current and previous links must differ")
|
||||||
}
|
}
|
||||||
for key, value := range s.Environment {
|
for label, path := range map[string]string{"caddy_config": s.CaddyConfig, "caddy_handler": s.CaddyHandler, "caddy_handler_template": s.CaddyHandlerTemplate} {
|
||||||
if !regexp.MustCompile(`^[A-Z][A-Z0-9_]{0,63}$`).MatchString(key) || strings.ContainsAny(value, "\x00\r\n") {
|
if err := safeAbsolute("deployment.singleton."+label, path); err != nil {
|
||||||
return fmt.Errorf("singleton environment entry %q is invalid", key)
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
if filepath.Clean(s.CaddyHandler) == filepath.Clean(s.CaddyHandlerTemplate) {
|
||||||
|
return errors.New("singleton Caddy handler and template must be different files")
|
||||||
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -9,13 +9,14 @@ import (
|
|||||||
|
|
||||||
func validConfig() Config {
|
func validConfig() Config {
|
||||||
return Config{
|
return Config{
|
||||||
SchemaVersion: 1,
|
SchemaVersion: 2,
|
||||||
Service: Service{Name: "example-site", AllowedHost: "example.test"},
|
Service: Service{Name: "example-site", AllowedHost: "example.test", EnvironmentFile: "/etc/tend/environment/example-site.env"},
|
||||||
Build: Build{Package: "./cmd/site", Binary: "example-site", Branch: "main"},
|
Build: Build{Package: "./cmd/site", Binary: "example-site", Branch: "main"},
|
||||||
Deployment: Deployment{
|
Deployment: Deployment{
|
||||||
Strategy: "blue_green", Root: "/opt/example-site", LockFile: "/run/lock/example-site.lock",
|
Strategy: "blue_green", Root: "/opt/example-site", LockFile: SharedLockFile,
|
||||||
StateFile: "/opt/example-site/state.json", HealthPath: "/healthz", ReadinessPath: "/readyz",
|
StateFile: "/opt/example-site/state.json", EventLog: "/opt/example-site/deployment-events.jsonl", HealthPath: "/healthz", ReadinessPath: "/readyz",
|
||||||
CandidateTimeoutSecs: 30, Smoke: []Smoke{{Path: "/", Contains: "Example"}},
|
CandidateTimeoutSecs: 30, ActivationWindowSecs: 10, Smoke: []Smoke{{Path: "/", Contains: "Example"}},
|
||||||
|
PublicSmoke: []PublicSmoke{{URL: "https://example.test/", Contains: "Example"}},
|
||||||
BlueGreen: &BlueGreen{
|
BlueGreen: &BlueGreen{
|
||||||
CaddyConfig: "/etc/caddy/Caddyfile", CaddyHandler: "/etc/caddy/example.caddy",
|
CaddyConfig: "/etc/caddy/Caddyfile", CaddyHandler: "/etc/caddy/example.caddy",
|
||||||
CaddyHandlerTemplate: "/etc/example/caddy.template",
|
CaddyHandlerTemplate: "/etc/example/caddy.template",
|
||||||
@@ -33,6 +34,24 @@ func TestValidateAcceptsBlueGreen(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestValidateSingletonRequiresDistinctCaddyHandoffFiles(t *testing.T) {
|
||||||
|
cfg := validConfig()
|
||||||
|
cfg.Deployment.Strategy = "singleton_candidate"
|
||||||
|
cfg.Deployment.BlueGreen = nil
|
||||||
|
cfg.Deployment.Singleton = &Singleton{
|
||||||
|
Unit: "example-site.service", Address: "127.0.0.1:8092", CandidateAddress: "127.0.0.1:18092", ListenEnv: "EXAMPLE_LISTEN",
|
||||||
|
CurrentLink: "/opt/example-site/current", PreviousLink: "/opt/example-site/previous", CaddyConfig: "/etc/caddy/Caddyfile",
|
||||||
|
CaddyHandler: "/etc/caddy/example-site.caddy", CaddyHandlerTemplate: "/etc/tend/caddy/example-site.template",
|
||||||
|
}
|
||||||
|
if err := cfg.Validate(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
cfg.Deployment.Singleton.CaddyHandlerTemplate = cfg.Deployment.Singleton.CaddyHandler
|
||||||
|
if err := cfg.Validate(); err == nil {
|
||||||
|
t.Fatal("expected shared handler/template path to be rejected")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestValidateRejectsHostileValues(t *testing.T) {
|
func TestValidateRejectsHostileValues(t *testing.T) {
|
||||||
tests := map[string]func(*Config){
|
tests := map[string]func(*Config){
|
||||||
"unknown strategy": func(c *Config) { c.Deployment.Strategy = "shell" },
|
"unknown strategy": func(c *Config) { c.Deployment.Strategy = "shell" },
|
||||||
@@ -41,6 +60,9 @@ func TestValidateRejectsHostileValues(t *testing.T) {
|
|||||||
"traversal": func(c *Config) { c.Build.Package = "./cmd/../secret" },
|
"traversal": func(c *Config) { c.Build.Package = "./cmd/../secret" },
|
||||||
"shared slot": func(c *Config) { c.Deployment.BlueGreen.Green.Link = c.Deployment.BlueGreen.Blue.Link },
|
"shared slot": func(c *Config) { c.Deployment.BlueGreen.Green.Link = c.Deployment.BlueGreen.Blue.Link },
|
||||||
"bad smoke": func(c *Config) { c.Deployment.Smoke[0].Path = "https://attacker.test/" },
|
"bad smoke": func(c *Config) { c.Deployment.Smoke[0].Path = "https://attacker.test/" },
|
||||||
|
"bad public smoke": func(c *Config) { c.Deployment.PublicSmoke[0].URL = "http://example.test/" },
|
||||||
|
"public secret query": func(c *Config) { c.Deployment.PublicSmoke[0].URL = "https://example.test/?token=secret" },
|
||||||
|
"environment sibling": func(c *Config) { c.Service.EnvironmentFile = "/etc/tend/environment-old/example.env" },
|
||||||
}
|
}
|
||||||
for name, mutate := range tests {
|
for name, mutate := range tests {
|
||||||
t.Run(name, func(t *testing.T) {
|
t.Run(name, func(t *testing.T) {
|
||||||
|
|||||||
+313
-76
@@ -14,6 +14,7 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gamertan.com/tend/internal/config"
|
"gamertan.com/tend/internal/config"
|
||||||
|
"gamertan.com/tend/internal/eventlog"
|
||||||
"gamertan.com/tend/internal/state"
|
"gamertan.com/tend/internal/state"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -29,6 +30,7 @@ type Report struct {
|
|||||||
Release string `json:"release,omitempty"`
|
Release string `json:"release,omitempty"`
|
||||||
ActiveRelease string `json:"active_release,omitempty"`
|
ActiveRelease string `json:"active_release,omitempty"`
|
||||||
PreviousRelease string `json:"previous_release,omitempty"`
|
PreviousRelease string `json:"previous_release,omitempty"`
|
||||||
|
EventWarnings int `json:"event_warnings,omitempty"`
|
||||||
}
|
}
|
||||||
type Status struct {
|
type Status struct {
|
||||||
State *state.Record `json:"state,omitempty"`
|
State *state.Record `json:"state,omitempty"`
|
||||||
@@ -41,10 +43,14 @@ type Manager struct {
|
|||||||
Now func() time.Time
|
Now func() time.Time
|
||||||
Prepare func(config.Config, string, string, string) (string, error)
|
Prepare func(config.Config, string, string, string) (string, error)
|
||||||
Inspect func(config.Config, string, string, string) error
|
Inspect func(config.Config, string, string, string) error
|
||||||
|
ReadIdentity func(string) (releaseIdentity, error)
|
||||||
|
OperationID func() (string, error)
|
||||||
|
AppendEvent func(string, eventlog.Event) error
|
||||||
|
Sleep func(context.Context, time.Duration) error
|
||||||
}
|
}
|
||||||
|
|
||||||
func NewManager(operator Operator) Manager {
|
func NewManager(operator Operator) Manager {
|
||||||
return Manager{Operator: operator, Now: time.Now, Prepare: prepareRelease, Inspect: inspectArtifact}
|
return Manager{Operator: operator, Now: time.Now, Prepare: prepareRelease, Inspect: inspectArtifact, ReadIdentity: readReleaseIdentity, OperationID: eventlog.OperationID, AppendEvent: eventlog.Append, Sleep: sleepContext}
|
||||||
}
|
}
|
||||||
|
|
||||||
func (m Manager) Deploy(ctx context.Context, cfg config.Config, request Request) (Report, error) {
|
func (m Manager) Deploy(ctx context.Context, cfg config.Config, request Request) (Report, error) {
|
||||||
@@ -66,10 +72,44 @@ func (m Manager) Deploy(ctx context.Context, cfg config.Config, request Request)
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return Report{}, err
|
return Report{}, err
|
||||||
}
|
}
|
||||||
|
started := m.Now()
|
||||||
|
eventWarnings := 0
|
||||||
|
identity, identityErr := m.ReadIdentity(release)
|
||||||
|
if identityErr != nil {
|
||||||
|
eventWarnings++
|
||||||
|
}
|
||||||
|
operationID := ""
|
||||||
|
if m.OperationID != nil {
|
||||||
|
operationID, err = m.OperationID()
|
||||||
|
if err != nil {
|
||||||
|
eventWarnings++
|
||||||
|
operationID = ""
|
||||||
|
}
|
||||||
|
}
|
||||||
|
emit := func(phase, slot, outcome string) {
|
||||||
|
if m.AppendEvent == nil || identityErr != nil || operationID == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
event := eventlog.Event{Version: eventlog.Version, OperationID: operationID, Service: cfg.Service.Name, ArtifactDigest: request.ApprovedSHA256, Commit: identity.Commit, ReleaseVersion: identity.Version, Phase: phase, Slot: slot, DurationMillis: max(0, m.Now().Sub(started).Milliseconds()), Outcome: outcome, ObservedAt: m.Now().UTC().Format(time.RFC3339Nano)}
|
||||||
|
if eventErr := m.AppendEvent(cfg.Deployment.EventLog, event); eventErr != nil {
|
||||||
|
eventWarnings++
|
||||||
|
}
|
||||||
|
}
|
||||||
record, err := loadOrBootstrap(cfg, m.Now())
|
record, err := loadOrBootstrap(cfg, m.Now())
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Report{}, err
|
return Report{}, err
|
||||||
}
|
}
|
||||||
|
attemptAt := m.Now().UTC().Format(time.RFC3339)
|
||||||
|
record.DesiredRelease = release
|
||||||
|
record.CandidateRelease = release
|
||||||
|
record.LastAttemptRelease = release
|
||||||
|
record.LastAttemptOutcome = "running"
|
||||||
|
record.LastAttemptAt = attemptAt
|
||||||
|
record.UpdatedAt = attemptAt
|
||||||
|
if err := state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, record); err != nil {
|
||||||
|
return Report{}, err
|
||||||
|
}
|
||||||
|
emit("candidate", inactiveSlot(cfg, record), "running")
|
||||||
switch cfg.Deployment.Strategy {
|
switch cfg.Deployment.Strategy {
|
||||||
case "blue_green":
|
case "blue_green":
|
||||||
err = m.deployBlueGreen(ctx, cfg, record, release)
|
err = m.deployBlueGreen(ctx, cfg, record, release)
|
||||||
@@ -79,13 +119,64 @@ func (m Manager) Deploy(ctx context.Context, cfg config.Config, request Request)
|
|||||||
err = errors.New("unsupported strategy")
|
err = errors.New("unsupported strategy")
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Report{}, err
|
failed := record
|
||||||
|
failed.CandidateRelease = ""
|
||||||
|
failed.LastAttemptOutcome = "failed"
|
||||||
|
failed.UpdatedAt = m.Now().UTC().Format(time.RFC3339)
|
||||||
|
_ = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, failed)
|
||||||
|
emit("activation", inactiveSlot(cfg, record), "failed")
|
||||||
|
return Report{EventWarnings: eventWarnings}, err
|
||||||
}
|
}
|
||||||
|
emit("activation", inactiveSlot(cfg, record), "succeeded")
|
||||||
updated, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
|
updated, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return Report{}, err
|
return Report{}, err
|
||||||
}
|
}
|
||||||
return Report{Validated: true, Mutation: "activated", Release: release, ActiveRelease: updated.ActiveRelease, PreviousRelease: updated.PreviousRelease}, nil
|
return Report{Validated: true, Mutation: "activated", Release: release, ActiveRelease: updated.ActiveRelease, PreviousRelease: updated.PreviousRelease, EventWarnings: eventWarnings}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type releaseIdentity struct {
|
||||||
|
Version string `json:"version"`
|
||||||
|
Commit string `json:"commit"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func readReleaseIdentity(release string) (releaseIdentity, error) {
|
||||||
|
b, err := os.ReadFile(filepath.Join(release, "RELEASE.json"))
|
||||||
|
if err != nil {
|
||||||
|
return releaseIdentity{}, fmt.Errorf("read installed release identity: %w", err)
|
||||||
|
}
|
||||||
|
if len(b) > 1<<20 {
|
||||||
|
return releaseIdentity{}, errors.New("installed release identity is too large")
|
||||||
|
}
|
||||||
|
var identity releaseIdentity
|
||||||
|
if err := json.Unmarshal(b, &identity); err != nil {
|
||||||
|
return releaseIdentity{}, errors.New("decode installed release identity")
|
||||||
|
}
|
||||||
|
if identity.Version == "" || identity.Commit == "" {
|
||||||
|
return releaseIdentity{}, errors.New("installed release identity is incomplete")
|
||||||
|
}
|
||||||
|
return identity, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func inactiveSlot(cfg config.Config, record state.Record) string {
|
||||||
|
if cfg.Deployment.Strategy == "singleton_candidate" {
|
||||||
|
return "singleton"
|
||||||
|
}
|
||||||
|
if record.ActiveSlot == "blue" {
|
||||||
|
return "green"
|
||||||
|
}
|
||||||
|
return "blue"
|
||||||
|
}
|
||||||
|
|
||||||
|
func sleepContext(ctx context.Context, duration time.Duration) error {
|
||||||
|
timer := time.NewTimer(duration)
|
||||||
|
defer timer.Stop()
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
return ctx.Err()
|
||||||
|
case <-timer.C:
|
||||||
|
return nil
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func loadOrBootstrap(cfg config.Config, now time.Time) (state.Record, error) {
|
func loadOrBootstrap(cfg config.Config, now time.Time) (state.Record, error) {
|
||||||
@@ -103,13 +194,13 @@ func loadOrBootstrap(cfg config.Config, now time.Time) (state.Record, error) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return state.Record{}, fmt.Errorf("bootstrap active slot: %w", err)
|
return state.Record{}, fmt.Errorf("bootstrap active slot: %w", err)
|
||||||
}
|
}
|
||||||
return state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, ActiveSlot: slot, ActiveRelease: release, UpdatedAt: now.UTC().Format(time.RFC3339)}, nil
|
return state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, DesiredRelease: release, ActiveSlot: slot, ActiveRelease: release, UpdatedAt: now.UTC().Format(time.RFC3339)}, nil
|
||||||
case "singleton_candidate":
|
case "singleton_candidate":
|
||||||
release, err := resolveReleaseLink(cfg.Deployment.Root, cfg.Deployment.Singleton.CurrentLink)
|
release, err := resolveReleaseLink(cfg.Deployment.Root, cfg.Deployment.Singleton.CurrentLink)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return state.Record{}, fmt.Errorf("bootstrap singleton: %w", err)
|
return state.Record{}, fmt.Errorf("bootstrap singleton: %w", err)
|
||||||
}
|
}
|
||||||
return state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, ActiveSlot: "singleton", ActiveRelease: release, UpdatedAt: now.UTC().Format(time.RFC3339)}, nil
|
return state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, DesiredRelease: release, ActiveSlot: "singleton", ActiveRelease: release, UpdatedAt: now.UTC().Format(time.RFC3339)}, nil
|
||||||
}
|
}
|
||||||
return state.Record{}, errors.New("unsupported strategy")
|
return state.Record{}, errors.New("unsupported strategy")
|
||||||
}
|
}
|
||||||
@@ -177,7 +268,14 @@ func (m Manager) deployBlueGreen(ctx context.Context, cfg config.Config, record
|
|||||||
if err = m.probeAll(ctx, cfg, slot.Address); err != nil {
|
if err = m.probeAll(ctx, cfg, slot.Address); err != nil {
|
||||||
return fmt.Errorf("post-activation smoke failed: %w", err)
|
return fmt.Errorf("post-activation smoke failed: %w", err)
|
||||||
}
|
}
|
||||||
next := state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, ActiveSlot: inactive, ActiveRelease: release, PreviousSlot: record.ActiveSlot, PreviousRelease: record.ActiveRelease, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
|
if err = m.probePublic(ctx, cfg, true); err != nil {
|
||||||
|
return fmt.Errorf("public-origin smoke failed: %w", err)
|
||||||
|
}
|
||||||
|
previous := slotConfig(bg, record.ActiveSlot)
|
||||||
|
if err = m.continuityWindow(ctx, cfg, previous.Address, true); err != nil {
|
||||||
|
return fmt.Errorf("activation continuity failed: %w", err)
|
||||||
|
}
|
||||||
|
next := state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, DesiredRelease: release, ActiveSlot: inactive, ActiveRelease: release, PreviousSlot: record.ActiveSlot, PreviousRelease: record.ActiveRelease, LastAttemptRelease: release, LastAttemptOutcome: "succeeded", LastAttemptAt: record.LastAttemptAt, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
|
||||||
if err = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next); err != nil {
|
if err = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -185,60 +283,10 @@ func (m Manager) deployBlueGreen(ctx context.Context, cfg config.Config, record
|
|||||||
}
|
}
|
||||||
|
|
||||||
func (m Manager) deploySingleton(ctx context.Context, cfg config.Config, record state.Record, release string) (err error) {
|
func (m Manager) deploySingleton(ctx context.Context, cfg config.Config, record state.Record, release string) (err error) {
|
||||||
single := *cfg.Deployment.Singleton
|
if err = m.activateSingletonRelease(ctx, cfg, release, true); err != nil {
|
||||||
candidateUnit := cfg.Service.Name + "-tend-candidate.service"
|
|
||||||
env := copyMap(single.Environment)
|
|
||||||
env[single.ListenEnv] = single.CandidateAddress
|
|
||||||
binary := filepath.Join(release, cfg.Build.Binary)
|
|
||||||
if err = m.Operator.StartCandidate(ctx, candidateUnit, binary, env); err != nil {
|
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
defer func() {
|
next := state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, DesiredRelease: release, ActiveSlot: "singleton", ActiveRelease: release, PreviousSlot: "singleton", PreviousRelease: record.ActiveRelease, LastAttemptRelease: release, LastAttemptOutcome: "succeeded", LastAttemptAt: record.LastAttemptAt, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
|
||||||
stopCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
|
||||||
defer cancel()
|
|
||||||
_ = m.Operator.Stop(stopCtx, candidateUnit)
|
|
||||||
}()
|
|
||||||
if err = m.probeAll(ctx, cfg, single.CandidateAddress); err != nil {
|
|
||||||
return fmt.Errorf("candidate failed: %w", err)
|
|
||||||
}
|
|
||||||
oldCurrent, err := resolveReleaseLink(cfg.Deployment.Root, single.CurrentLink)
|
|
||||||
if err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
oldPrevious, previousErr := resolveReleaseLink(cfg.Deployment.Root, single.PreviousLink)
|
|
||||||
currentChanged := false
|
|
||||||
previousChanged := false
|
|
||||||
defer func() {
|
|
||||||
if err == nil {
|
|
||||||
return
|
|
||||||
}
|
|
||||||
if currentChanged {
|
|
||||||
_ = replaceSymlink(single.CurrentLink, oldCurrent)
|
|
||||||
_ = m.Operator.Restart(ctx, single.Unit)
|
|
||||||
}
|
|
||||||
if previousChanged {
|
|
||||||
if previousErr == nil {
|
|
||||||
_ = replaceSymlink(single.PreviousLink, oldPrevious)
|
|
||||||
} else {
|
|
||||||
_ = removeSymlink(single.PreviousLink)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}()
|
|
||||||
if err = replaceSymlink(single.PreviousLink, oldCurrent); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
previousChanged = true
|
|
||||||
if err = replaceSymlink(single.CurrentLink, release); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
currentChanged = true
|
|
||||||
if err = m.Operator.Restart(ctx, single.Unit); err != nil {
|
|
||||||
return err
|
|
||||||
}
|
|
||||||
if err = m.probeAll(ctx, cfg, single.Address); err != nil {
|
|
||||||
return fmt.Errorf("post-activation smoke failed: %w", err)
|
|
||||||
}
|
|
||||||
next := state.Record{SchemaVersion: 1, Strategy: cfg.Deployment.Strategy, ActiveSlot: "singleton", ActiveRelease: release, PreviousSlot: "singleton", PreviousRelease: record.ActiveRelease, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
|
|
||||||
if err = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next); err != nil {
|
if err = state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
@@ -258,6 +306,21 @@ func (m Manager) Rollback(ctx context.Context, cfg config.Config) (state.Record,
|
|||||||
if record.PreviousRelease == "" {
|
if record.PreviousRelease == "" {
|
||||||
return state.Record{}, errors.New("no previous release is recorded")
|
return state.Record{}, errors.New("no previous release is recorded")
|
||||||
}
|
}
|
||||||
|
started := m.Now()
|
||||||
|
identity, identityErr := m.ReadIdentity(record.PreviousRelease)
|
||||||
|
digest, digestErr := releaseDigest(record.PreviousRelease)
|
||||||
|
operationID := ""
|
||||||
|
if m.OperationID != nil {
|
||||||
|
operationID, _ = m.OperationID()
|
||||||
|
}
|
||||||
|
emit := func(outcome string) {
|
||||||
|
if m.AppendEvent == nil || identityErr != nil || digestErr != nil || operationID == "" {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
event := eventlog.Event{Version: eventlog.Version, OperationID: operationID, Service: cfg.Service.Name, ArtifactDigest: digest, Commit: identity.Commit, ReleaseVersion: identity.Version, Phase: "rollback", Slot: record.PreviousSlot, DurationMillis: max(0, m.Now().Sub(started).Milliseconds()), Outcome: outcome, ObservedAt: m.Now().UTC().Format(time.RFC3339Nano)}
|
||||||
|
_ = m.AppendEvent(cfg.Deployment.EventLog, event)
|
||||||
|
}
|
||||||
|
emit("running")
|
||||||
switch cfg.Deployment.Strategy {
|
switch cfg.Deployment.Strategy {
|
||||||
case "blue_green":
|
case "blue_green":
|
||||||
err = m.rollbackBlueGreen(ctx, cfg, record)
|
err = m.rollbackBlueGreen(ctx, cfg, record)
|
||||||
@@ -267,10 +330,29 @@ func (m Manager) Rollback(ctx context.Context, cfg config.Config) (state.Record,
|
|||||||
err = errors.New("unsupported strategy")
|
err = errors.New("unsupported strategy")
|
||||||
}
|
}
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
emit("failed")
|
||||||
return state.Record{}, err
|
return state.Record{}, err
|
||||||
}
|
}
|
||||||
|
emit("succeeded")
|
||||||
return state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
|
return state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func releaseDigest(release string) (string, error) {
|
||||||
|
name := filepath.Base(release)
|
||||||
|
if !strings.HasPrefix(name, "sha256-") {
|
||||||
|
return "", errors.New("release is not content addressed")
|
||||||
|
}
|
||||||
|
digest := strings.TrimPrefix(name, "sha256-")
|
||||||
|
if len(digest) != 64 {
|
||||||
|
return "", errors.New("release digest is invalid")
|
||||||
|
}
|
||||||
|
for _, character := range digest {
|
||||||
|
if !strings.ContainsRune("0123456789abcdef", character) {
|
||||||
|
return "", errors.New("release digest is invalid")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return digest, nil
|
||||||
|
}
|
||||||
func (m Manager) rollbackBlueGreen(ctx context.Context, cfg config.Config, record state.Record) (err error) {
|
func (m Manager) rollbackBlueGreen(ctx context.Context, cfg config.Config, record state.Record) (err error) {
|
||||||
bg := *cfg.Deployment.BlueGreen
|
bg := *cfg.Deployment.BlueGreen
|
||||||
slot := slotConfig(bg, record.PreviousSlot)
|
slot := slotConfig(bg, record.PreviousSlot)
|
||||||
@@ -313,33 +395,151 @@ func (m Manager) rollbackBlueGreen(ctx context.Context, cfg config.Config, recor
|
|||||||
if err = m.probeHealthReadiness(ctx, cfg, slot.Address); err != nil {
|
if err = m.probeHealthReadiness(ctx, cfg, slot.Address); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
next := state.Record{SchemaVersion: 1, Strategy: record.Strategy, ActiveSlot: record.PreviousSlot, ActiveRelease: record.PreviousRelease, PreviousSlot: record.ActiveSlot, PreviousRelease: record.ActiveRelease, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
|
if err = m.probePublic(ctx, cfg, false); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
next := state.Record{SchemaVersion: 1, Strategy: record.Strategy, DesiredRelease: record.PreviousRelease, ActiveSlot: record.PreviousSlot, ActiveRelease: record.PreviousRelease, PreviousSlot: record.ActiveSlot, PreviousRelease: record.ActiveRelease, LastAttemptRelease: record.PreviousRelease, LastAttemptOutcome: "rolled_back", LastAttemptAt: m.Now().UTC().Format(time.RFC3339), UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
|
||||||
return state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next)
|
return state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next)
|
||||||
}
|
}
|
||||||
func (m Manager) rollbackSingleton(ctx context.Context, cfg config.Config, record state.Record) (err error) {
|
func (m Manager) rollbackSingleton(ctx context.Context, cfg config.Config, record state.Record) (err error) {
|
||||||
|
if err = m.activateSingletonRelease(ctx, cfg, record.PreviousRelease, false); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
next := state.Record{SchemaVersion: 1, Strategy: record.Strategy, DesiredRelease: record.PreviousRelease, ActiveSlot: "singleton", ActiveRelease: record.PreviousRelease, PreviousSlot: "singleton", PreviousRelease: record.ActiveRelease, LastAttemptRelease: record.PreviousRelease, LastAttemptOutcome: "rolled_back", LastAttemptAt: m.Now().UTC().Format(time.RFC3339), UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
|
||||||
|
return state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next)
|
||||||
|
}
|
||||||
|
|
||||||
|
// activateSingletonRelease keeps public traffic on a proven process while the
|
||||||
|
// installed fixed-address unit changes release. The transient candidate first
|
||||||
|
// receives traffic, remains healthy through the handoff, and is stopped only
|
||||||
|
// after Caddy points back to the verified installed unit.
|
||||||
|
func (m Manager) activateSingletonRelease(ctx context.Context, cfg config.Config, release string, checkMarkers bool) (err error) {
|
||||||
single := *cfg.Deployment.Singleton
|
single := *cfg.Deployment.Singleton
|
||||||
|
candidateUnit := cfg.Service.Name + "-tend-candidate.service"
|
||||||
|
env := map[string]string{single.ListenEnv: single.CandidateAddress}
|
||||||
|
binary := filepath.Join(release, cfg.Build.Binary)
|
||||||
|
if err = m.Operator.StartCandidate(ctx, candidateUnit, binary, cfg.Service.EnvironmentFile, env); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
stopCandidate := true
|
||||||
|
defer func() {
|
||||||
|
if !stopCandidate {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
stopCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
_ = m.Operator.Stop(stopCtx, candidateUnit)
|
||||||
|
}()
|
||||||
|
probeLocal := m.probeHealthReadiness
|
||||||
|
if checkMarkers {
|
||||||
|
probeLocal = m.probeAll
|
||||||
|
}
|
||||||
|
if err = probeLocal(ctx, cfg, single.CandidateAddress); err != nil {
|
||||||
|
return fmt.Errorf("candidate failed: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
oldHandler, err := os.ReadFile(single.CaddyHandler)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("read current Caddy handler: %w", err)
|
||||||
|
}
|
||||||
oldCurrent, err := resolveReleaseLink(cfg.Deployment.Root, single.CurrentLink)
|
oldCurrent, err := resolveReleaseLink(cfg.Deployment.Root, single.CurrentLink)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err = replaceSymlink(single.CurrentLink, record.PreviousRelease); err != nil {
|
oldPrevious, previousErr := resolveReleaseLink(cfg.Deployment.Root, single.PreviousLink)
|
||||||
return err
|
handlerChanged := false
|
||||||
}
|
currentChanged := false
|
||||||
|
previousChanged := false
|
||||||
defer func() {
|
defer func() {
|
||||||
if err != nil {
|
if err == nil {
|
||||||
_ = replaceSymlink(single.CurrentLink, oldCurrent)
|
return
|
||||||
_ = m.Operator.Restart(ctx, single.Unit)
|
}
|
||||||
|
recoveryErr := error(nil)
|
||||||
|
if currentChanged {
|
||||||
|
if restoreErr := replaceSymlink(single.CurrentLink, oldCurrent); restoreErr != nil {
|
||||||
|
recoveryErr = errors.Join(recoveryErr, restoreErr)
|
||||||
|
} else if restoreErr = m.Operator.Restart(ctx, single.Unit); restoreErr != nil {
|
||||||
|
recoveryErr = errors.Join(recoveryErr, restoreErr)
|
||||||
|
} else if restoreErr = m.probeHealthReadiness(ctx, cfg, single.Address); restoreErr != nil {
|
||||||
|
recoveryErr = errors.Join(recoveryErr, restoreErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if previousChanged {
|
||||||
|
var restoreErr error
|
||||||
|
if previousErr == nil {
|
||||||
|
restoreErr = replaceSymlink(single.PreviousLink, oldPrevious)
|
||||||
|
} else {
|
||||||
|
restoreErr = removeSymlink(single.PreviousLink)
|
||||||
|
}
|
||||||
|
recoveryErr = errors.Join(recoveryErr, restoreErr)
|
||||||
|
}
|
||||||
|
if handlerChanged && recoveryErr == nil {
|
||||||
|
if restoreErr := atomicWrite(single.CaddyHandler, oldHandler, 0o644); restoreErr != nil {
|
||||||
|
recoveryErr = errors.Join(recoveryErr, restoreErr)
|
||||||
|
} else if restoreErr = m.Operator.ValidateCaddy(ctx, single.CaddyConfig); restoreErr != nil {
|
||||||
|
recoveryErr = errors.Join(recoveryErr, restoreErr)
|
||||||
|
} else if restoreErr = m.Operator.ReloadCaddy(ctx); restoreErr != nil {
|
||||||
|
recoveryErr = errors.Join(recoveryErr, restoreErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if recoveryErr != nil && handlerChanged {
|
||||||
|
stopCandidate = false
|
||||||
|
err = errors.Join(err, fmt.Errorf("singleton recovery incomplete; candidate remains routed for operator recovery: %w", recoveryErr))
|
||||||
}
|
}
|
||||||
}()
|
}()
|
||||||
|
|
||||||
|
candidateHandler, err := renderHandler(single.CaddyHandlerTemplate, single.CandidateAddress)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if err = atomicWrite(single.CaddyHandler, candidateHandler, 0o644); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
handlerChanged = true
|
||||||
|
if err = m.Operator.ValidateCaddy(ctx, single.CaddyConfig); err != nil {
|
||||||
|
return fmt.Errorf("candidate Caddy validation failed: %w", err)
|
||||||
|
}
|
||||||
|
if err = m.Operator.ReloadCaddy(ctx); err != nil {
|
||||||
|
return fmt.Errorf("candidate Caddy reload failed: %w", err)
|
||||||
|
}
|
||||||
|
if err = m.probePublic(ctx, cfg, checkMarkers); err != nil {
|
||||||
|
return fmt.Errorf("candidate public-origin smoke failed: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err = replaceSymlink(single.PreviousLink, oldCurrent); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
previousChanged = true
|
||||||
|
if err = replaceSymlink(single.CurrentLink, release); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
currentChanged = true
|
||||||
if err = m.Operator.Restart(ctx, single.Unit); err != nil {
|
if err = m.Operator.Restart(ctx, single.Unit); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
if err = m.probeHealthReadiness(ctx, cfg, single.Address); err != nil {
|
if err = probeLocal(ctx, cfg, single.Address); err != nil {
|
||||||
|
return fmt.Errorf("post-activation smoke failed: %w", err)
|
||||||
|
}
|
||||||
|
installedHandler, err := renderHandler(single.CaddyHandlerTemplate, single.Address)
|
||||||
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
_ = replaceSymlink(single.PreviousLink, record.ActiveRelease)
|
if err = atomicWrite(single.CaddyHandler, installedHandler, 0o644); err != nil {
|
||||||
next := state.Record{SchemaVersion: 1, Strategy: record.Strategy, ActiveSlot: "singleton", ActiveRelease: record.PreviousRelease, PreviousSlot: "singleton", PreviousRelease: record.ActiveRelease, UpdatedAt: m.Now().UTC().Format(time.RFC3339)}
|
return err
|
||||||
return state.Store(cfg.Deployment.StateFile, cfg.Deployment.Root, next)
|
}
|
||||||
|
if err = m.Operator.ValidateCaddy(ctx, single.CaddyConfig); err != nil {
|
||||||
|
return fmt.Errorf("installed Caddy validation failed: %w", err)
|
||||||
|
}
|
||||||
|
if err = m.Operator.ReloadCaddy(ctx); err != nil {
|
||||||
|
return fmt.Errorf("installed Caddy reload failed: %w", err)
|
||||||
|
}
|
||||||
|
if err = m.probePublic(ctx, cfg, checkMarkers); err != nil {
|
||||||
|
return fmt.Errorf("public-origin smoke failed: %w", err)
|
||||||
|
}
|
||||||
|
if err = m.continuityWindow(ctx, cfg, single.CandidateAddress, checkMarkers); err != nil {
|
||||||
|
return fmt.Errorf("activation continuity failed: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func (m Manager) Status(ctx context.Context, cfg config.Config) (Status, error) {
|
func (m Manager) Status(ctx context.Context, cfg config.Config) (Status, error) {
|
||||||
@@ -435,6 +635,50 @@ func (m Manager) probeHealthReadiness(ctx context.Context, cfg config.Config, ad
|
|||||||
return m.probe(ctx, cfg, address, checks)
|
return m.probe(ctx, cfg, address, checks)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (m Manager) probePublic(ctx context.Context, cfg config.Config, checkMarkers bool) error {
|
||||||
|
timeout := time.Duration(cfg.Deployment.CandidateTimeoutSecs) * time.Second
|
||||||
|
for _, check := range cfg.Deployment.PublicSmoke {
|
||||||
|
attempt, cancel := context.WithTimeout(ctx, timeout)
|
||||||
|
contains := check.Contains
|
||||||
|
if !checkMarkers {
|
||||||
|
contains = ""
|
||||||
|
}
|
||||||
|
err := m.Operator.ProbeURL(attempt, check.URL, contains)
|
||||||
|
cancel()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (m Manager) continuityWindow(ctx context.Context, cfg config.Config, previousAddress string, checkMarkers bool) error {
|
||||||
|
steps := cfg.Deployment.ActivationWindowSecs * 4
|
||||||
|
if steps < 1 {
|
||||||
|
steps = 1
|
||||||
|
}
|
||||||
|
for step := 0; step < steps; step++ {
|
||||||
|
if err := m.probePublic(ctx, cfg, checkMarkers); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if previousAddress != "" {
|
||||||
|
if err := m.probeHealthReadiness(ctx, cfg, previousAddress); err != nil {
|
||||||
|
return fmt.Errorf("previous slot lost continuity: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if step+1 < steps {
|
||||||
|
sleep := m.Sleep
|
||||||
|
if sleep == nil {
|
||||||
|
sleep = sleepContext
|
||||||
|
}
|
||||||
|
if err := sleep(ctx, 250*time.Millisecond); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func (m Manager) probe(ctx context.Context, cfg config.Config, address string, checks []config.Smoke) error {
|
func (m Manager) probe(ctx context.Context, cfg config.Config, address string, checks []config.Smoke) error {
|
||||||
timeout := time.Duration(cfg.Deployment.CandidateTimeoutSecs) * time.Second
|
timeout := time.Duration(cfg.Deployment.CandidateTimeoutSecs) * time.Second
|
||||||
for _, check := range checks {
|
for _, check := range checks {
|
||||||
@@ -465,13 +709,6 @@ func slotConfig(bg config.BlueGreen, name string) config.Slot {
|
|||||||
}
|
}
|
||||||
return bg.Green
|
return bg.Green
|
||||||
}
|
}
|
||||||
func copyMap(source map[string]string) map[string]string {
|
|
||||||
target := make(map[string]string, len(source)+1)
|
|
||||||
for k, v := range source {
|
|
||||||
target[k] = v
|
|
||||||
}
|
|
||||||
return target
|
|
||||||
}
|
|
||||||
func resolveReleaseLink(root, link string) (string, error) {
|
func resolveReleaseLink(root, link string) (string, error) {
|
||||||
info, err := os.Lstat(link)
|
info, err := os.Lstat(link)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
+283
-21
@@ -5,6 +5,7 @@ package deploy
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -12,21 +13,33 @@ import (
|
|||||||
"time"
|
"time"
|
||||||
|
|
||||||
"gamertan.com/tend/internal/config"
|
"gamertan.com/tend/internal/config"
|
||||||
|
"gamertan.com/tend/internal/eventlog"
|
||||||
"gamertan.com/tend/internal/state"
|
"gamertan.com/tend/internal/state"
|
||||||
)
|
)
|
||||||
|
|
||||||
type fakeOperator struct {
|
type fakeOperator struct {
|
||||||
failReload bool
|
failReload bool
|
||||||
|
failReloadAt int
|
||||||
|
reloads int
|
||||||
failRestartUnit string
|
failRestartUnit string
|
||||||
|
failRestartOnce bool
|
||||||
rejectMarkers bool
|
rejectMarkers bool
|
||||||
active map[string]bool
|
active map[string]bool
|
||||||
starts, stops, restarts []string
|
starts, stops, restarts []string
|
||||||
probes []string
|
probes []string
|
||||||
|
publicProbes []string
|
||||||
|
failPublic bool
|
||||||
|
failPublicAfter int
|
||||||
|
candidateEnvironment map[string]string
|
||||||
|
candidateFile string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (f *fakeOperator) Restart(_ context.Context, unit string) error {
|
func (f *fakeOperator) Restart(_ context.Context, unit string) error {
|
||||||
f.restarts = append(f.restarts, unit)
|
f.restarts = append(f.restarts, unit)
|
||||||
if unit == f.failRestartUnit {
|
if unit == f.failRestartUnit {
|
||||||
|
if f.failRestartOnce {
|
||||||
|
f.failRestartUnit = ""
|
||||||
|
}
|
||||||
return errors.New("injected restart failure")
|
return errors.New("injected restart failure")
|
||||||
}
|
}
|
||||||
f.active[unit] = true
|
f.active[unit] = true
|
||||||
@@ -40,17 +53,92 @@ func (f *fakeOperator) Stop(_ context.Context, unit string) error {
|
|||||||
func (f *fakeOperator) IsActive(_ context.Context, unit string) (bool, error) {
|
func (f *fakeOperator) IsActive(_ context.Context, unit string) (bool, error) {
|
||||||
return f.active[unit], nil
|
return f.active[unit], nil
|
||||||
}
|
}
|
||||||
func (f *fakeOperator) StartCandidate(_ context.Context, unit, binary string, env map[string]string) error {
|
func (f *fakeOperator) StartCandidate(_ context.Context, unit, binary, environmentFile string, env map[string]string) error {
|
||||||
if !filepath.IsAbs(binary) || len(env) == 0 {
|
if !filepath.IsAbs(binary) || !filepath.IsAbs(environmentFile) || len(env) == 0 {
|
||||||
return errors.New("bad candidate")
|
return errors.New("bad candidate")
|
||||||
}
|
}
|
||||||
f.starts = append(f.starts, unit)
|
f.starts = append(f.starts, unit)
|
||||||
|
f.candidateFile = environmentFile
|
||||||
|
f.candidateEnvironment = make(map[string]string, len(env))
|
||||||
|
for key, value := range env {
|
||||||
|
f.candidateEnvironment[key] = value
|
||||||
|
}
|
||||||
f.active[unit] = true
|
f.active[unit] = true
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
func (f *fakeOperator) ProbeURL(_ context.Context, value, contains string) error {
|
||||||
|
f.publicProbes = append(f.publicProbes, value)
|
||||||
|
if f.failPublic || (f.failPublicAfter > 0 && len(f.publicProbes) >= f.failPublicAfter) {
|
||||||
|
return errors.New("injected public smoke failure")
|
||||||
|
}
|
||||||
|
if f.rejectMarkers && contains != "" {
|
||||||
|
return errors.New("unexpected future-release smoke marker")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPublicSmokeFailureRestoresBlueGreenHandlerAndSlot(t *testing.T) {
|
||||||
|
cfg, old, fresh := baseConfig(t, "blue_green")
|
||||||
|
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
|
||||||
|
template := filepath.Join(cfg.Deployment.Root, "handler.template")
|
||||||
|
original := []byte("reverse_proxy 127.0.0.1:8090\n")
|
||||||
|
_ = os.WriteFile(handler, original, 0o644)
|
||||||
|
_ = os.WriteFile(template, []byte("reverse_proxy {{UPSTREAM}}\n"), 0o644)
|
||||||
|
blue := filepath.Join(cfg.Deployment.Root, "slots", "blue")
|
||||||
|
green := filepath.Join(cfg.Deployment.Root, "slots", "green")
|
||||||
|
_ = replaceSymlink(blue, old)
|
||||||
|
_ = replaceSymlink(green, old)
|
||||||
|
cfg.Deployment.BlueGreen = &config.BlueGreen{CaddyConfig: filepath.Join(cfg.Deployment.Root, "Caddyfile"), CaddyHandler: handler, CaddyHandlerTemplate: template, BootstrapActive: "blue", Blue: config.Slot{Unit: "example-blue.service", Address: "127.0.0.1:8090", Link: blue}, Green: config.Slot{Unit: "example-green.service", Address: "127.0.0.1:8091", Link: green}}
|
||||||
|
operator := &fakeOperator{active: map[string]bool{}, failPublic: true}
|
||||||
|
if _, err := manager(operator, fresh).Deploy(context.Background(), cfg, Request{Activate: true}); err == nil {
|
||||||
|
t.Fatal("expected public smoke failure")
|
||||||
|
}
|
||||||
|
body, _ := os.ReadFile(handler)
|
||||||
|
if string(body) != string(original) {
|
||||||
|
t.Fatalf("handler not restored: %q", body)
|
||||||
|
}
|
||||||
|
target, err := resolveReleaseLink(cfg.Deployment.Root, green)
|
||||||
|
if err != nil || target != old {
|
||||||
|
t.Fatalf("green=%q err=%v", target, err)
|
||||||
|
}
|
||||||
|
record, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if record.ActiveRelease != old || record.LastAttemptOutcome != "failed" || record.CandidateRelease != "" || record.LastAttemptRelease != fresh {
|
||||||
|
t.Fatalf("failed attempt state=%+v", record)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestContinuityFailureRestoresBlueGreenRoute(t *testing.T) {
|
||||||
|
cfg, old, fresh := baseConfig(t, "blue_green")
|
||||||
|
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
|
||||||
|
template := filepath.Join(cfg.Deployment.Root, "handler.template")
|
||||||
|
original := []byte("reverse_proxy 127.0.0.1:8090\n")
|
||||||
|
_ = os.WriteFile(handler, original, 0o644)
|
||||||
|
_ = os.WriteFile(template, []byte("reverse_proxy {{UPSTREAM}}\n"), 0o644)
|
||||||
|
blue := filepath.Join(cfg.Deployment.Root, "slots", "blue")
|
||||||
|
green := filepath.Join(cfg.Deployment.Root, "slots", "green")
|
||||||
|
_ = replaceSymlink(blue, old)
|
||||||
|
_ = replaceSymlink(green, old)
|
||||||
|
cfg.Deployment.BlueGreen = &config.BlueGreen{CaddyConfig: filepath.Join(cfg.Deployment.Root, "Caddyfile"), CaddyHandler: handler, CaddyHandlerTemplate: template, BootstrapActive: "blue", Blue: config.Slot{Unit: "example-blue.service", Address: "127.0.0.1:8090", Link: blue}, Green: config.Slot{Unit: "example-green.service", Address: "127.0.0.1:8091", Link: green}}
|
||||||
|
operator := &fakeOperator{active: map[string]bool{}, failPublicAfter: 3}
|
||||||
|
if _, err := manager(operator, fresh).Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)}); err == nil || !strings.Contains(err.Error(), "continuity") {
|
||||||
|
t.Fatalf("expected continuity failure, got %v", err)
|
||||||
|
}
|
||||||
|
body, _ := os.ReadFile(handler)
|
||||||
|
if string(body) != string(original) {
|
||||||
|
t.Fatalf("handler not restored: %q", body)
|
||||||
|
}
|
||||||
|
target, err := resolveReleaseLink(cfg.Deployment.Root, green)
|
||||||
|
if err != nil || target != old {
|
||||||
|
t.Fatalf("green=%q err=%v", target, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
func (f *fakeOperator) ValidateCaddy(context.Context, string) error { return nil }
|
func (f *fakeOperator) ValidateCaddy(context.Context, string) error { return nil }
|
||||||
func (f *fakeOperator) ReloadCaddy(context.Context) error {
|
func (f *fakeOperator) ReloadCaddy(context.Context) error {
|
||||||
if f.failReload {
|
f.reloads++
|
||||||
|
if f.failReload || (f.failReloadAt > 0 && f.reloads == f.failReloadAt) {
|
||||||
return errors.New("injected reload failure")
|
return errors.New("injected reload failure")
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -69,7 +157,7 @@ func baseConfig(t *testing.T, strategy string) (config.Config, string, string) {
|
|||||||
if err := os.MkdirAll(filepath.Join(root, "releases"), 0o755); err != nil {
|
if err := os.MkdirAll(filepath.Join(root, "releases"), 0o755); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
old := filepath.Join(root, "releases", "legacy-old")
|
old := filepath.Join(root, "releases", "sha256-"+strings.Repeat("c", 64))
|
||||||
fresh := filepath.Join(root, "releases", "sha256-"+strings.Repeat("a", 64))
|
fresh := filepath.Join(root, "releases", "sha256-"+strings.Repeat("a", 64))
|
||||||
for _, dir := range []string{old, fresh} {
|
for _, dir := range []string{old, fresh} {
|
||||||
if err := os.Mkdir(dir, 0o755); err != nil {
|
if err := os.Mkdir(dir, 0o755); err != nil {
|
||||||
@@ -79,11 +167,34 @@ func baseConfig(t *testing.T, strategy string) (config.Config, string, string) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
cfg := config.Config{SchemaVersion: 1, Service: config.Service{Name: "example-site", AllowedHost: "example.test"}, Build: config.Build{Package: "./cmd/site", Binary: "app", Branch: "main"}, Deployment: config.Deployment{Strategy: strategy, Root: root, LockFile: filepath.Join(root, "deploy.lock"), StateFile: filepath.Join(root, "state.json"), HealthPath: "/healthz", ReadinessPath: "/readyz", CandidateTimeoutSecs: 2, Smoke: []config.Smoke{{Path: "/", Contains: "Example"}}}}
|
cfg := config.Config{SchemaVersion: 2, Service: config.Service{Name: "example-site", AllowedHost: "example.test", EnvironmentFile: "/etc/tend/environment/example-site.env"}, Build: config.Build{Package: "./cmd/site", Binary: "app", Branch: "main"}, Deployment: config.Deployment{Strategy: strategy, Root: root, LockFile: filepath.Join(root, "deploy.lock"), StateFile: filepath.Join(root, "state.json"), EventLog: filepath.Join(root, "deployment-events.jsonl"), HealthPath: "/healthz", ReadinessPath: "/readyz", CandidateTimeoutSecs: 2, ActivationWindowSecs: 1, Smoke: []config.Smoke{{Path: "/", Contains: "Example"}}, PublicSmoke: []config.PublicSmoke{{URL: "https://example.test/", Contains: "Example"}}}}
|
||||||
return cfg, old, fresh
|
return cfg, old, fresh
|
||||||
}
|
}
|
||||||
func manager(operator Operator, fresh string) Manager {
|
func manager(operator Operator, fresh string) Manager {
|
||||||
return Manager{Operator: operator, Now: func() time.Time { return time.Unix(100, 0).UTC() }, Prepare: func(config.Config, string, string, string) (string, error) { return fresh, nil }, Inspect: func(config.Config, string, string, string) error { return nil }}
|
return Manager{Operator: operator, Now: func() time.Time { return time.Unix(100, 0).UTC() }, Prepare: func(config.Config, string, string, string) (string, error) { return fresh, nil }, Inspect: func(config.Config, string, string, string) error { return nil }, ReadIdentity: func(string) (releaseIdentity, error) {
|
||||||
|
return releaseIdentity{Version: "v0.2.0-preview.1", Commit: strings.Repeat("b", 40)}, nil
|
||||||
|
}, OperationID: func() (string, error) { return strings.Repeat("d", 32), nil }, Sleep: func(context.Context, time.Duration) error { return nil }}
|
||||||
|
}
|
||||||
|
|
||||||
|
func singletonSettings(t *testing.T, cfg config.Config, currentRelease string) (*config.Singleton, string) {
|
||||||
|
t.Helper()
|
||||||
|
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
|
||||||
|
template := filepath.Join(cfg.Deployment.Root, "handler.template")
|
||||||
|
if err := os.WriteFile(handler, []byte("reverse_proxy 127.0.0.1:8092\n"), 0o640); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(template, []byte("reverse_proxy {{UPSTREAM}}\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
current := filepath.Join(cfg.Deployment.Root, "current")
|
||||||
|
if err := replaceSymlink(current, currentRelease); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return &config.Singleton{
|
||||||
|
Unit: "example-site.service", Address: "127.0.0.1:8092", CandidateAddress: "127.0.0.1:18092", ListenEnv: "EXAMPLE_LISTEN",
|
||||||
|
CurrentLink: current, PreviousLink: filepath.Join(cfg.Deployment.Root, "previous"), CaddyConfig: filepath.Join(cfg.Deployment.Root, "Caddyfile"),
|
||||||
|
CaddyHandler: handler, CaddyHandlerTemplate: template,
|
||||||
|
}, handler
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestBlueGreenActivationAndRollback(t *testing.T) {
|
func TestBlueGreenActivationAndRollback(t *testing.T) {
|
||||||
@@ -110,13 +221,28 @@ func TestBlueGreenActivationAndRollback(t *testing.T) {
|
|||||||
}
|
}
|
||||||
operator := &fakeOperator{active: map[string]bool{"example-blue.service": true, "example-green.service": true}}
|
operator := &fakeOperator{active: map[string]bool{"example-blue.service": true, "example-green.service": true}}
|
||||||
m := manager(operator, fresh)
|
m := manager(operator, fresh)
|
||||||
report, err := m.Deploy(context.Background(), cfg, Request{Activate: true})
|
var events []eventlog.Event
|
||||||
|
m.AppendEvent = func(_ string, event eventlog.Event) error {
|
||||||
|
events = append(events, event)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
report, err := m.Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if report.ActiveRelease != fresh || report.PreviousRelease != old {
|
if report.ActiveRelease != fresh || report.PreviousRelease != old {
|
||||||
t.Fatalf("report=%+v", report)
|
t.Fatalf("report=%+v", report)
|
||||||
}
|
}
|
||||||
|
deployed, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if deployed.DesiredRelease != fresh || deployed.CandidateRelease != "" || deployed.LastAttemptRelease != fresh || deployed.LastAttemptOutcome != "succeeded" {
|
||||||
|
t.Fatalf("deployment identity state=%+v", deployed)
|
||||||
|
}
|
||||||
|
if len(events) != 2 || events[0].Phase != "candidate" || events[0].Outcome != "running" || events[1].Phase != "activation" || events[1].Outcome != "succeeded" || events[0].OperationID != events[1].OperationID {
|
||||||
|
t.Fatalf("events=%+v", events)
|
||||||
|
}
|
||||||
if info, err := os.Stat(handler); err != nil || info.Mode().Perm() != 0o640 {
|
if info, err := os.Stat(handler); err != nil || info.Mode().Perm() != 0o640 {
|
||||||
t.Fatalf("handler mode=%v err=%v", info.Mode().Perm(), err)
|
t.Fatalf("handler mode=%v err=%v", info.Mode().Perm(), err)
|
||||||
}
|
}
|
||||||
@@ -132,6 +258,9 @@ func TestBlueGreenActivationAndRollback(t *testing.T) {
|
|||||||
if record.ActiveRelease != old || record.PreviousRelease != fresh {
|
if record.ActiveRelease != old || record.PreviousRelease != fresh {
|
||||||
t.Fatalf("rollback=%+v", record)
|
t.Fatalf("rollback=%+v", record)
|
||||||
}
|
}
|
||||||
|
if len(events) != 4 || events[2].Phase != "rollback" || events[2].Outcome != "running" || events[3].Phase != "rollback" || events[3].Outcome != "succeeded" || events[2].OperationID != events[3].OperationID || events[2].ArtifactDigest != strings.Repeat("c", 64) {
|
||||||
|
t.Fatalf("rollback events=%+v", events)
|
||||||
|
}
|
||||||
if len(operator.probes) != 4 {
|
if len(operator.probes) != 4 {
|
||||||
t.Fatalf("rollback probes=%#v", operator.probes)
|
t.Fatalf("rollback probes=%#v", operator.probes)
|
||||||
}
|
}
|
||||||
@@ -142,6 +271,58 @@ func TestBlueGreenActivationAndRollback(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestDeploymentEvidenceCanNeverBlockActivationOrRollback(t *testing.T) {
|
||||||
|
cfg, old, fresh := baseConfig(t, "singleton_candidate")
|
||||||
|
cfg.Deployment.Singleton, _ = singletonSettings(t, cfg, old)
|
||||||
|
m := manager(&fakeOperator{active: map[string]bool{}}, fresh)
|
||||||
|
appendCalls := 0
|
||||||
|
m.AppendEvent = func(string, eventlog.Event) error {
|
||||||
|
appendCalls++
|
||||||
|
return errors.New("injected event failure")
|
||||||
|
}
|
||||||
|
report, err := m.Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if report.EventWarnings != 2 || report.ActiveRelease != fresh {
|
||||||
|
t.Fatalf("report=%+v", report)
|
||||||
|
}
|
||||||
|
if _, err := m.Rollback(context.Background(), cfg); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
record, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
|
||||||
|
if err != nil || record.ActiveRelease != old || appendCalls != 4 {
|
||||||
|
t.Fatalf("record=%+v appends=%d err=%v", record, appendCalls, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestDeploymentEvidenceIdentityAndEntropyAreBestEffort(t *testing.T) {
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
damage func(*Manager)
|
||||||
|
}{
|
||||||
|
{"identity", func(manager *Manager) {
|
||||||
|
manager.ReadIdentity = func(string) (releaseIdentity, error) {
|
||||||
|
return releaseIdentity{}, errors.New("injected identity failure")
|
||||||
|
}
|
||||||
|
}},
|
||||||
|
{"entropy", func(manager *Manager) {
|
||||||
|
manager.OperationID = func() (string, error) { return "", errors.New("injected entropy failure") }
|
||||||
|
}},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
cfg, old, fresh := baseConfig(t, "singleton_candidate")
|
||||||
|
cfg.Deployment.Singleton, _ = singletonSettings(t, cfg, old)
|
||||||
|
manager := manager(&fakeOperator{active: map[string]bool{}}, fresh)
|
||||||
|
test.damage(&manager)
|
||||||
|
report, err := manager.Deploy(context.Background(), cfg, Request{Activate: true, ApprovedSHA256: strings.Repeat("a", 64)})
|
||||||
|
if err != nil || report.EventWarnings != 1 || report.ActiveRelease != fresh {
|
||||||
|
t.Fatalf("report=%+v err=%v", report, err)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestBlueGreenCaddyFailureRestoresHandlerAndSlot(t *testing.T) {
|
func TestBlueGreenCaddyFailureRestoresHandlerAndSlot(t *testing.T) {
|
||||||
cfg, old, fresh := baseConfig(t, "blue_green")
|
cfg, old, fresh := baseConfig(t, "blue_green")
|
||||||
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
|
handler := filepath.Join(cfg.Deployment.Root, "handler.caddy")
|
||||||
@@ -167,37 +348,47 @@ func TestBlueGreenCaddyFailureRestoresHandlerAndSlot(t *testing.T) {
|
|||||||
if err != nil || target != old {
|
if err != nil || target != old {
|
||||||
t.Fatalf("green=%q err=%v", target, err)
|
t.Fatalf("green=%q err=%v", target, err)
|
||||||
}
|
}
|
||||||
if _, err := os.Stat(cfg.Deployment.StateFile); !os.IsNotExist(err) {
|
failed, err := state.Load(cfg.Deployment.StateFile, cfg.Deployment.Root, cfg.Deployment.Strategy)
|
||||||
t.Fatal("failed activation wrote state")
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if failed.ActiveRelease != old || failed.LastAttemptOutcome != "failed" || failed.CandidateRelease != "" {
|
||||||
|
t.Fatalf("failed state=%+v", failed)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestSingletonRestartFailureRestoresPointers(t *testing.T) {
|
func TestSingletonRestartFailureRestoresPointers(t *testing.T) {
|
||||||
cfg, old, fresh := baseConfig(t, "singleton_candidate")
|
cfg, old, fresh := baseConfig(t, "singleton_candidate")
|
||||||
previous := filepath.Join(cfg.Deployment.Root, "previous")
|
settings, handler := singletonSettings(t, cfg, old)
|
||||||
current := filepath.Join(cfg.Deployment.Root, "current")
|
cfg.Deployment.Singleton = settings
|
||||||
_ = replaceSymlink(current, old)
|
operator := &fakeOperator{active: map[string]bool{}, failRestartUnit: "example-site.service", failRestartOnce: true}
|
||||||
cfg.Deployment.Singleton = &config.Singleton{Unit: "example-site.service", Address: "127.0.0.1:8092", CandidateAddress: "127.0.0.1:18092", ListenEnv: "EXAMPLE_LISTEN", CurrentLink: current, PreviousLink: previous}
|
|
||||||
operator := &fakeOperator{active: map[string]bool{}, failRestartUnit: "example-site.service"}
|
|
||||||
m := manager(operator, fresh)
|
m := manager(operator, fresh)
|
||||||
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err == nil {
|
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err == nil {
|
||||||
t.Fatal("expected failure")
|
t.Fatal("expected failure")
|
||||||
}
|
}
|
||||||
target, err := resolveReleaseLink(cfg.Deployment.Root, current)
|
target, err := resolveReleaseLink(cfg.Deployment.Root, cfg.Deployment.Singleton.CurrentLink)
|
||||||
if err != nil || target != old {
|
if err != nil || target != old {
|
||||||
t.Fatalf("current=%q err=%v", target, err)
|
t.Fatalf("current=%q err=%v", target, err)
|
||||||
}
|
}
|
||||||
if _, err := os.Lstat(previous); !os.IsNotExist(err) {
|
if _, err := os.Lstat(cfg.Deployment.Singleton.PreviousLink); !os.IsNotExist(err) {
|
||||||
t.Fatal("previous pointer was not restored")
|
t.Fatal("previous pointer was not restored")
|
||||||
}
|
}
|
||||||
|
body, err := os.ReadFile(handler)
|
||||||
|
if err != nil || string(body) != "reverse_proxy 127.0.0.1:8092\n" {
|
||||||
|
t.Fatalf("handler=%q err=%v", body, err)
|
||||||
|
}
|
||||||
|
if info, err := os.Stat(handler); err != nil || info.Mode().Perm() != 0o640 {
|
||||||
|
t.Fatalf("handler mode=%v err=%v", info.Mode().Perm(), err)
|
||||||
|
}
|
||||||
|
if operator.active["example-site-tend-candidate.service"] {
|
||||||
|
t.Fatal("candidate was not stopped after successful restoration")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestStatePersistsOnlyAfterSuccessfulActivation(t *testing.T) {
|
func TestStateRecordsSuccessfulActivation(t *testing.T) {
|
||||||
cfg, old, fresh := baseConfig(t, "singleton_candidate")
|
cfg, old, fresh := baseConfig(t, "singleton_candidate")
|
||||||
current := filepath.Join(cfg.Deployment.Root, "current")
|
settings, handler := singletonSettings(t, cfg, old)
|
||||||
previous := filepath.Join(cfg.Deployment.Root, "previous")
|
cfg.Deployment.Singleton = settings
|
||||||
_ = replaceSymlink(current, old)
|
|
||||||
cfg.Deployment.Singleton = &config.Singleton{Unit: "example-site.service", Address: "127.0.0.1:8092", CandidateAddress: "127.0.0.1:18092", ListenEnv: "EXAMPLE_LISTEN", CurrentLink: current, PreviousLink: previous}
|
|
||||||
operator := &fakeOperator{active: map[string]bool{}}
|
operator := &fakeOperator{active: map[string]bool{}}
|
||||||
m := manager(operator, fresh)
|
m := manager(operator, fresh)
|
||||||
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err != nil {
|
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err != nil {
|
||||||
@@ -210,4 +401,75 @@ func TestStatePersistsOnlyAfterSuccessfulActivation(t *testing.T) {
|
|||||||
if record.ActiveRelease != fresh || record.PreviousRelease != old {
|
if record.ActiveRelease != fresh || record.PreviousRelease != old {
|
||||||
t.Fatalf("state=%+v", record)
|
t.Fatalf("state=%+v", record)
|
||||||
}
|
}
|
||||||
|
if operator.candidateFile != cfg.Service.EnvironmentFile || len(operator.candidateEnvironment) != 1 || operator.candidateEnvironment["EXAMPLE_LISTEN"] != "127.0.0.1:18092" {
|
||||||
|
t.Fatalf("candidate file=%q environment=%#v", operator.candidateFile, operator.candidateEnvironment)
|
||||||
|
}
|
||||||
|
body, err := os.ReadFile(handler)
|
||||||
|
if err != nil || string(body) != "reverse_proxy 127.0.0.1:8092\n" {
|
||||||
|
t.Fatalf("handler=%q err=%v", body, err)
|
||||||
|
}
|
||||||
|
if operator.reloads != 2 || operator.active["example-site-tend-candidate.service"] {
|
||||||
|
t.Fatalf("reloads=%d active=%#v", operator.reloads, operator.active)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSingletonContinuityFailureRestoresHandlerPointersAndService(t *testing.T) {
|
||||||
|
cfg, old, fresh := baseConfig(t, "singleton_candidate")
|
||||||
|
settings, handler := singletonSettings(t, cfg, old)
|
||||||
|
cfg.Deployment.Singleton = settings
|
||||||
|
operator := &fakeOperator{active: map[string]bool{"example-site.service": true}, failPublicAfter: 4}
|
||||||
|
m := manager(operator, fresh)
|
||||||
|
if _, err := m.Deploy(context.Background(), cfg, Request{Activate: true}); err == nil || !strings.Contains(err.Error(), "continuity") {
|
||||||
|
t.Fatalf("expected continuity failure, got %v", err)
|
||||||
|
}
|
||||||
|
current, err := resolveReleaseLink(cfg.Deployment.Root, cfg.Deployment.Singleton.CurrentLink)
|
||||||
|
if err != nil || current != old {
|
||||||
|
t.Fatalf("current=%q err=%v", current, err)
|
||||||
|
}
|
||||||
|
body, err := os.ReadFile(handler)
|
||||||
|
if err != nil || string(body) != "reverse_proxy 127.0.0.1:8092\n" {
|
||||||
|
t.Fatalf("handler=%q err=%v", body, err)
|
||||||
|
}
|
||||||
|
if operator.active["example-site-tend-candidate.service"] {
|
||||||
|
t.Fatal("candidate was not stopped after continuity restoration")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSingletonCaddyReloadFailuresRestorePriorRoute(t *testing.T) {
|
||||||
|
for _, reload := range []int{1, 2} {
|
||||||
|
t.Run(fmt.Sprintf("reload-%d", reload), func(t *testing.T) {
|
||||||
|
cfg, old, fresh := baseConfig(t, "singleton_candidate")
|
||||||
|
settings, handler := singletonSettings(t, cfg, old)
|
||||||
|
cfg.Deployment.Singleton = settings
|
||||||
|
operator := &fakeOperator{active: map[string]bool{"example-site.service": true}, failReloadAt: reload}
|
||||||
|
if _, err := manager(operator, fresh).Deploy(context.Background(), cfg, Request{Activate: true}); err == nil || !strings.Contains(err.Error(), "Caddy reload failed") {
|
||||||
|
t.Fatalf("expected Caddy reload failure, got %v", err)
|
||||||
|
}
|
||||||
|
current, err := resolveReleaseLink(cfg.Deployment.Root, settings.CurrentLink)
|
||||||
|
if err != nil || current != old {
|
||||||
|
t.Fatalf("current=%q err=%v", current, err)
|
||||||
|
}
|
||||||
|
body, err := os.ReadFile(handler)
|
||||||
|
if err != nil || string(body) != "reverse_proxy 127.0.0.1:8092\n" {
|
||||||
|
t.Fatalf("handler=%q err=%v", body, err)
|
||||||
|
}
|
||||||
|
if operator.active["example-site-tend-candidate.service"] {
|
||||||
|
t.Fatal("candidate was not stopped after route restoration")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestSingletonIncompleteRecoveryKeepsProvenCandidateRunning(t *testing.T) {
|
||||||
|
cfg, old, fresh := baseConfig(t, "singleton_candidate")
|
||||||
|
settings, _ := singletonSettings(t, cfg, old)
|
||||||
|
cfg.Deployment.Singleton = settings
|
||||||
|
operator := &fakeOperator{active: map[string]bool{"example-site.service": true}, failReload: true}
|
||||||
|
_, err := manager(operator, fresh).Deploy(context.Background(), cfg, Request{Activate: true})
|
||||||
|
if err == nil || !strings.Contains(err.Error(), "candidate remains routed for operator recovery") {
|
||||||
|
t.Fatalf("expected explicit incomplete recovery, got %v", err)
|
||||||
|
}
|
||||||
|
if !operator.active["example-site-tend-candidate.service"] {
|
||||||
|
t.Fatal("proven candidate was stopped despite incomplete route restoration")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
//go:build linux
|
||||||
|
|
||||||
|
package deploy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"path/filepath"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestHostWideLockSerializesIndependentServices(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "tend-deploy.lock")
|
||||||
|
first, err := acquireLock(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
defer first.Close()
|
||||||
|
if second, err := acquireLock(path); err == nil {
|
||||||
|
_ = second.Close()
|
||||||
|
t.Fatal("second service acquired the shared activation lock")
|
||||||
|
}
|
||||||
|
if err = first.Close(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
third, err := acquireLock(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
_ = third.Close()
|
||||||
|
}
|
||||||
@@ -20,10 +20,11 @@ type Operator interface {
|
|||||||
Restart(context.Context, string) error
|
Restart(context.Context, string) error
|
||||||
Stop(context.Context, string) error
|
Stop(context.Context, string) error
|
||||||
IsActive(context.Context, string) (bool, error)
|
IsActive(context.Context, string) (bool, error)
|
||||||
StartCandidate(context.Context, string, string, map[string]string) error
|
StartCandidate(context.Context, string, string, string, map[string]string) error
|
||||||
ValidateCaddy(context.Context, string) error
|
ValidateCaddy(context.Context, string) error
|
||||||
ReloadCaddy(context.Context) error
|
ReloadCaddy(context.Context) error
|
||||||
Probe(context.Context, string, string, string, string) error
|
Probe(context.Context, string, string, string, string) error
|
||||||
|
ProbeURL(context.Context, string, string) error
|
||||||
}
|
}
|
||||||
|
|
||||||
type SystemOperator struct {
|
type SystemOperator struct {
|
||||||
@@ -49,7 +50,7 @@ func (o SystemOperator) IsActive(ctx context.Context, unit string) (bool, error)
|
|||||||
}
|
}
|
||||||
return strings.TrimSpace(string(out)) == "active", nil
|
return strings.TrimSpace(string(out)) == "active", nil
|
||||||
}
|
}
|
||||||
func (o SystemOperator) StartCandidate(ctx context.Context, unit, binary string, env map[string]string) error {
|
func (o SystemOperator) StartCandidate(ctx context.Context, unit, binary, environmentFile string, env map[string]string) error {
|
||||||
args := []string{
|
args := []string{
|
||||||
"--unit", unit, "--collect",
|
"--unit", unit, "--collect",
|
||||||
"--property=DynamicUser=yes", "--property=NoNewPrivileges=yes",
|
"--property=DynamicUser=yes", "--property=NoNewPrivileges=yes",
|
||||||
@@ -63,6 +64,7 @@ func (o SystemOperator) StartCandidate(ctx context.Context, unit, binary string,
|
|||||||
"--property=RestrictSUIDSGID=yes", "--property=LockPersonality=yes",
|
"--property=RestrictSUIDSGID=yes", "--property=LockPersonality=yes",
|
||||||
"--property=MemoryDenyWriteExecute=yes", "--property=CapabilityBoundingSet=",
|
"--property=MemoryDenyWriteExecute=yes", "--property=CapabilityBoundingSet=",
|
||||||
"--property=AmbientCapabilities=",
|
"--property=AmbientCapabilities=",
|
||||||
|
"--property=EnvironmentFile=" + environmentFile,
|
||||||
}
|
}
|
||||||
keys := make([]string, 0, len(env))
|
keys := make([]string, 0, len(env))
|
||||||
for key := range env {
|
for key := range env {
|
||||||
@@ -86,11 +88,23 @@ func (o SystemOperator) ReloadCaddy(ctx context.Context) error {
|
|||||||
}
|
}
|
||||||
func (o SystemOperator) Probe(ctx context.Context, address, host, path, contains string) error {
|
func (o SystemOperator) Probe(ctx context.Context, address, host, path, contains string) error {
|
||||||
u := url.URL{Scheme: "http", Host: address, Path: path}
|
u := url.URL{Scheme: "http", Host: address, Path: path}
|
||||||
req, err := http.NewRequestWithContext(ctx, http.MethodGet, u.String(), nil)
|
return o.probeRequest(ctx, u.String(), host, contains)
|
||||||
|
}
|
||||||
|
func (o SystemOperator) ProbeURL(ctx context.Context, value, contains string) error {
|
||||||
|
u, err := url.Parse(value)
|
||||||
|
if err != nil || u.Scheme != "https" || u.Host == "" || u.User != nil || u.Fragment != "" {
|
||||||
|
return errors.New("public probe URL is invalid")
|
||||||
|
}
|
||||||
|
return o.probeRequest(ctx, u.String(), "", contains)
|
||||||
|
}
|
||||||
|
func (o SystemOperator) probeRequest(ctx context.Context, value, host, contains string) error {
|
||||||
|
req, err := http.NewRequestWithContext(ctx, http.MethodGet, value, nil)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
if host != "" {
|
||||||
req.Host = host
|
req.Host = host
|
||||||
|
}
|
||||||
client := &http.Client{Timeout: o.Timeout, CheckRedirect: func(*http.Request, []*http.Request) error { return errors.New("redirect refused") }}
|
client := &http.Client{Timeout: o.Timeout, CheckRedirect: func(*http.Request, []*http.Request) error { return errors.New("redirect refused") }}
|
||||||
response, err := client.Do(req)
|
response, err := client.Do(req)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -5,6 +5,7 @@ package deploy
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"reflect"
|
"reflect"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
)
|
)
|
||||||
|
|
||||||
@@ -23,13 +24,13 @@ func TestStartCandidateUsesArgumentVectorAndHardenedUnit(t *testing.T) {
|
|||||||
runner := &recordingRunner{}
|
runner := &recordingRunner{}
|
||||||
operator := SystemOperator{Runner: runner}
|
operator := SystemOperator{Runner: runner}
|
||||||
env := map[string]string{"Z_ENV": "safe value", "A_ENV": "first"}
|
env := map[string]string{"Z_ENV": "safe value", "A_ENV": "first"}
|
||||||
if err := operator.StartCandidate(context.Background(), "example-tend-candidate.service", "/opt/example/releases/sha256-a/app", env); err != nil {
|
if err := operator.StartCandidate(context.Background(), "example-tend-candidate.service", "/opt/example/releases/sha256-a/app", "/etc/tend/environment/example.env", env); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if runner.name != "systemd-run" {
|
if runner.name != "systemd-run" {
|
||||||
t.Fatalf("command=%q", runner.name)
|
t.Fatalf("command=%q", runner.name)
|
||||||
}
|
}
|
||||||
required := []string{"--property=DynamicUser=yes", "--property=NoNewPrivileges=yes", "--property=ProtectSystem=strict", "--property=MemoryDenyWriteExecute=yes", "--property=CapabilityBoundingSet=", "--setenv", "A_ENV=first", "--setenv", "Z_ENV=safe value", "--", "/opt/example/releases/sha256-a/app"}
|
required := []string{"--property=DynamicUser=yes", "--property=NoNewPrivileges=yes", "--property=ProtectSystem=strict", "--property=MemoryDenyWriteExecute=yes", "--property=CapabilityBoundingSet=", "--property=EnvironmentFile=/etc/tend/environment/example.env", "--setenv", "A_ENV=first", "--setenv", "Z_ENV=safe value", "--", "/opt/example/releases/sha256-a/app"}
|
||||||
cursor := 0
|
cursor := 0
|
||||||
for _, arg := range runner.args {
|
for _, arg := range runner.args {
|
||||||
if cursor < len(required) && arg == required[cursor] {
|
if cursor < len(required) && arg == required[cursor] {
|
||||||
@@ -42,4 +43,7 @@ func TestStartCandidateUsesArgumentVectorAndHardenedUnit(t *testing.T) {
|
|||||||
if reflect.DeepEqual(runner.args, []string{"sh", "-c"}) {
|
if reflect.DeepEqual(runner.args, []string{"sh", "-c"}) {
|
||||||
t.Fatal("candidate command used a shell")
|
t.Fatal("candidate command used a shell")
|
||||||
}
|
}
|
||||||
|
if strings.Contains(strings.Join(runner.args, "\n"), "SUPER_SECRET") {
|
||||||
|
t.Fatal("candidate arguments exposed a secret value")
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -196,6 +196,15 @@ func extractArtifact(artifact, stage string) error {
|
|||||||
_ = out.Close()
|
_ = out.Close()
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
|
// OpenFile modes are filtered through the caller's umask. Tend is
|
||||||
|
// commonly invoked by a root account with umask 0077, while release
|
||||||
|
// binaries must remain executable by their dedicated service users.
|
||||||
|
// Reapply the validated, name-derived mode explicitly before the file
|
||||||
|
// becomes part of an immutable release.
|
||||||
|
if err := out.Chmod(mode); err != nil {
|
||||||
|
_ = out.Close()
|
||||||
|
return err
|
||||||
|
}
|
||||||
if err := out.Sync(); err != nil {
|
if err := out.Sync(); err != nil {
|
||||||
_ = out.Close()
|
_ = out.Close()
|
||||||
return err
|
return err
|
||||||
|
|||||||
@@ -0,0 +1,75 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
//go:build linux
|
||||||
|
|
||||||
|
package deploy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"archive/tar"
|
||||||
|
"compress/gzip"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"syscall"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestExtractArtifactAppliesReleaseModesUnderRestrictiveUmask(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
artifact := filepath.Join(dir, "release.tar.gz")
|
||||||
|
file, err := os.OpenFile(artifact, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
gz := gzip.NewWriter(file)
|
||||||
|
tw := tar.NewWriter(gz)
|
||||||
|
entries := map[string][]byte{
|
||||||
|
"bundle/app": []byte("executable"),
|
||||||
|
"bundle/BUILDINFO.json": []byte("{}"),
|
||||||
|
"bundle/RELEASE.json": []byte("{}"),
|
||||||
|
"bundle/SBOM.spdx.json": []byte("{}"),
|
||||||
|
"bundle/SHA256SUMS": []byte("checksums"),
|
||||||
|
}
|
||||||
|
for name, body := range entries {
|
||||||
|
if err := tw.WriteHeader(&tar.Header{Name: name, Typeflag: tar.TypeReg, Mode: 0o600, Size: int64(len(body))}); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if _, err := tw.Write(body); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if err := tw.Close(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := gz.Close(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := file.Close(); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
oldUmask := syscall.Umask(0o077)
|
||||||
|
t.Cleanup(func() { syscall.Umask(oldUmask) })
|
||||||
|
stage := filepath.Join(dir, "stage")
|
||||||
|
if err := os.Mkdir(stage, 0o700); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := extractArtifact(artifact, stage); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for name, want := range map[string]os.FileMode{
|
||||||
|
"app": 0o755,
|
||||||
|
"BUILDINFO.json": 0o644,
|
||||||
|
"RELEASE.json": 0o644,
|
||||||
|
"SBOM.spdx.json": 0o644,
|
||||||
|
"SHA256SUMS": 0o644,
|
||||||
|
} {
|
||||||
|
info, err := os.Stat(filepath.Join(stage, name))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if got := info.Mode().Perm(); got != want {
|
||||||
|
t.Fatalf("%s mode=%#o want=%#o", name, got, want)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,115 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
package eventlog
|
||||||
|
|
||||||
|
import (
|
||||||
|
"crypto/rand"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
const Version = 1
|
||||||
|
|
||||||
|
var safeValue = regexp.MustCompile(`^[A-Za-z0-9._:/@+-]{1,256}$`)
|
||||||
|
var hexDigest = regexp.MustCompile(`^[0-9a-f]{64}$`)
|
||||||
|
var gitCommit = regexp.MustCompile(`^[0-9a-f]{40}$`)
|
||||||
|
var operationID = regexp.MustCompile(`^[0-9a-f]{32}$`)
|
||||||
|
|
||||||
|
type Event struct {
|
||||||
|
Version int `json:"version"`
|
||||||
|
OperationID string `json:"operation_id"`
|
||||||
|
Service string `json:"service"`
|
||||||
|
ArtifactDigest string `json:"artifact_digest"`
|
||||||
|
Commit string `json:"commit"`
|
||||||
|
ReleaseVersion string `json:"release_version"`
|
||||||
|
Phase string `json:"phase"`
|
||||||
|
Slot string `json:"slot,omitempty"`
|
||||||
|
DurationMillis int64 `json:"duration_ms"`
|
||||||
|
Outcome string `json:"outcome"`
|
||||||
|
ObservedAt string `json:"observed_at"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func OperationID() (string, error) {
|
||||||
|
b := make([]byte, 16)
|
||||||
|
if _, err := rand.Read(b); err != nil {
|
||||||
|
return "", errors.New("cryptographic randomness unavailable")
|
||||||
|
}
|
||||||
|
return hex.EncodeToString(b), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func Append(path string, event Event) error {
|
||||||
|
if err := event.validate(); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(path) || filepath.Clean(path) != path {
|
||||||
|
return errors.New("event log path must be absolute and clean")
|
||||||
|
}
|
||||||
|
if err := os.MkdirAll(filepath.Dir(path), 0o750); err != nil {
|
||||||
|
return fmt.Errorf("create event directory: %w", err)
|
||||||
|
}
|
||||||
|
if info, err := os.Lstat(path); err == nil {
|
||||||
|
if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 || info.Mode().Perm()&0o022 != 0 {
|
||||||
|
return errors.New("event log must be a non-writable regular non-symlink file")
|
||||||
|
}
|
||||||
|
} else if !errors.Is(err, os.ErrNotExist) {
|
||||||
|
return fmt.Errorf("inspect event log: %w", err)
|
||||||
|
}
|
||||||
|
b, err := json.Marshal(event)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("encode deployment event: %w", err)
|
||||||
|
}
|
||||||
|
if len(b) > 4096 {
|
||||||
|
return errors.New("deployment event exceeds bound")
|
||||||
|
}
|
||||||
|
b = append(b, '\n')
|
||||||
|
fd, err := syscall.Open(path, syscall.O_WRONLY|syscall.O_APPEND|syscall.O_CREAT|syscall.O_CLOEXEC|syscall.O_NOFOLLOW, 0o640)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("open event log: %w", err)
|
||||||
|
}
|
||||||
|
file := os.NewFile(uintptr(fd), path)
|
||||||
|
if file == nil {
|
||||||
|
_ = syscall.Close(fd)
|
||||||
|
return errors.New("open event log file")
|
||||||
|
}
|
||||||
|
defer file.Close()
|
||||||
|
n, err := file.Write(b)
|
||||||
|
if err != nil || n != len(b) {
|
||||||
|
return errors.New("write complete deployment event")
|
||||||
|
}
|
||||||
|
if err := file.Sync(); err != nil {
|
||||||
|
return fmt.Errorf("sync deployment event: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (e Event) validate() error {
|
||||||
|
if e.Version != Version || !operationID.MatchString(e.OperationID) {
|
||||||
|
return errors.New("deployment event identity is invalid")
|
||||||
|
}
|
||||||
|
if !hexDigest.MatchString(e.ArtifactDigest) || !gitCommit.MatchString(e.Commit) {
|
||||||
|
return errors.New("deployment event provenance is invalid")
|
||||||
|
}
|
||||||
|
for label, value := range map[string]string{"service": e.Service, "artifact_digest": e.ArtifactDigest, "commit": e.Commit, "release_version": e.ReleaseVersion, "phase": e.Phase, "outcome": e.Outcome} {
|
||||||
|
if !safeValue.MatchString(value) || strings.ContainsRune(value, '\x00') {
|
||||||
|
return fmt.Errorf("deployment event %s is invalid", label)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if e.Slot != "" && !safeValue.MatchString(e.Slot) {
|
||||||
|
return errors.New("deployment event slot is invalid")
|
||||||
|
}
|
||||||
|
if e.DurationMillis < 0 {
|
||||||
|
return errors.New("deployment event duration is invalid")
|
||||||
|
}
|
||||||
|
if _, err := time.Parse(time.RFC3339Nano, e.ObservedAt); err != nil {
|
||||||
|
return errors.New("deployment event timestamp is invalid")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
package eventlog
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
"time"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestAppendBoundedEvent(t *testing.T) {
|
||||||
|
path := filepath.Join(t.TempDir(), "events.jsonl")
|
||||||
|
id, err := OperationID()
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
event := Event{Version: 1, OperationID: id, Service: "site", ArtifactDigest: strings.Repeat("a", 64), Commit: strings.Repeat("b", 40), ReleaseVersion: "v0.2.0-preview.1", Phase: "activation", Slot: "green", Outcome: "succeeded", ObservedAt: time.Now().UTC().Format(time.RFC3339Nano)}
|
||||||
|
if err := Append(path, event); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
b, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var decoded Event
|
||||||
|
if err := json.Unmarshal(b, &decoded); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if decoded.OperationID != id || strings.Contains(string(b), "secret") {
|
||||||
|
t.Fatalf("unexpected event: %s", b)
|
||||||
|
}
|
||||||
|
if info, _ := os.Stat(path); info.Mode().Perm() != 0o640 {
|
||||||
|
t.Fatalf("mode=%04o", info.Mode().Perm())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestAppendRejectsUnboundedValuesAndSymlink(t *testing.T) {
|
||||||
|
id, _ := OperationID()
|
||||||
|
event := Event{Version: 1, OperationID: id, Service: "site\nsecret", ArtifactDigest: "digest", Commit: "commit", ReleaseVersion: "version", Phase: "activation", Outcome: "failed", ObservedAt: time.Now().UTC().Format(time.RFC3339Nano)}
|
||||||
|
if err := Append(filepath.Join(t.TempDir(), "events.jsonl"), event); err == nil {
|
||||||
|
t.Fatal("expected unsafe value rejection")
|
||||||
|
}
|
||||||
|
dir := t.TempDir()
|
||||||
|
target := filepath.Join(dir, "target")
|
||||||
|
if err := os.WriteFile(target, nil, 0o640); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
link := filepath.Join(dir, "events.jsonl")
|
||||||
|
if err := os.Symlink(target, link); err != nil {
|
||||||
|
t.Skip(err)
|
||||||
|
}
|
||||||
|
event.Service = "site"
|
||||||
|
if err := Append(link, event); err == nil {
|
||||||
|
t.Fatal("expected symlink rejection")
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
//go:build linux
|
||||||
|
|
||||||
|
package serverpolicy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"syscall"
|
||||||
|
)
|
||||||
|
|
||||||
|
func rootOwned(info os.FileInfo) bool {
|
||||||
|
stat, ok := info.Sys().(*syscall.Stat_t)
|
||||||
|
return ok && stat.Uid == 0
|
||||||
|
}
|
||||||
@@ -0,0 +1,8 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
//go:build !linux
|
||||||
|
|
||||||
|
package serverpolicy
|
||||||
|
|
||||||
|
import "os"
|
||||||
|
|
||||||
|
func rootOwned(os.FileInfo) bool { return false }
|
||||||
@@ -0,0 +1,300 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
// Package serverpolicy validates the root-owned allowlist used by Tend's
|
||||||
|
// restricted SSH receiver. It contains service names and paths, never secrets.
|
||||||
|
package serverpolicy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"sort"
|
||||||
|
"strings"
|
||||||
|
|
||||||
|
"gamertan.com/tend/internal/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
const SchemaVersion = 1
|
||||||
|
|
||||||
|
var servicePattern = regexp.MustCompile(`^[a-z][a-z0-9-]{1,62}$`)
|
||||||
|
|
||||||
|
type Policy struct {
|
||||||
|
SchemaVersion int `json:"schema_version"`
|
||||||
|
ConfigRoot string `json:"config_root"`
|
||||||
|
IncomingRoot string `json:"incoming_root"`
|
||||||
|
SharedLockFile string `json:"shared_lock_file"`
|
||||||
|
Services map[string]ServicePolicy `json:"services"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type ServicePolicy struct {
|
||||||
|
Config string `json:"config"`
|
||||||
|
MaxArtifactBytes int64 `json:"max_artifact_bytes"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type CheckedService struct {
|
||||||
|
Name string
|
||||||
|
Config config.Config
|
||||||
|
Policy ServicePolicy
|
||||||
|
}
|
||||||
|
|
||||||
|
func Parse(reader io.Reader) (Policy, error) {
|
||||||
|
limited := io.LimitReader(reader, 1<<20+1)
|
||||||
|
body, err := io.ReadAll(limited)
|
||||||
|
if err != nil {
|
||||||
|
return Policy{}, err
|
||||||
|
}
|
||||||
|
if len(body) > 1<<20 {
|
||||||
|
return Policy{}, errors.New("server policy exceeds 1 MiB")
|
||||||
|
}
|
||||||
|
decoder := json.NewDecoder(bytes.NewReader(body))
|
||||||
|
decoder.DisallowUnknownFields()
|
||||||
|
var policy Policy
|
||||||
|
if err = decoder.Decode(&policy); err != nil {
|
||||||
|
return Policy{}, fmt.Errorf("decode server policy: %w", err)
|
||||||
|
}
|
||||||
|
var trailing any
|
||||||
|
if err = decoder.Decode(&trailing); !errors.Is(err, io.EOF) {
|
||||||
|
return Policy{}, errors.New("server policy contains trailing data")
|
||||||
|
}
|
||||||
|
if err = policy.Validate(); err != nil {
|
||||||
|
return Policy{}, err
|
||||||
|
}
|
||||||
|
return policy, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func Load(path string) (Policy, error) {
|
||||||
|
if err := secureDirectory(filepath.Dir(path), 0); err != nil {
|
||||||
|
return Policy{}, fmt.Errorf("server policy directory: %w", err)
|
||||||
|
}
|
||||||
|
if err := secureFile(path, 0o600); err != nil {
|
||||||
|
return Policy{}, fmt.Errorf("server policy: %w", err)
|
||||||
|
}
|
||||||
|
file, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return Policy{}, err
|
||||||
|
}
|
||||||
|
defer file.Close()
|
||||||
|
return Parse(file)
|
||||||
|
}
|
||||||
|
|
||||||
|
func (policy Policy) Validate() error {
|
||||||
|
if policy.SchemaVersion != SchemaVersion {
|
||||||
|
return fmt.Errorf("server policy schema_version must be %d", SchemaVersion)
|
||||||
|
}
|
||||||
|
if policy.ConfigRoot != "/etc/tend/services" {
|
||||||
|
return errors.New("server policy config_root must be /etc/tend/services")
|
||||||
|
}
|
||||||
|
if policy.IncomingRoot != "/var/lib/tend/incoming" {
|
||||||
|
return errors.New("server policy incoming_root must be /var/lib/tend/incoming")
|
||||||
|
}
|
||||||
|
if policy.SharedLockFile != config.SharedLockFile {
|
||||||
|
return fmt.Errorf("server policy shared_lock_file must be %s", config.SharedLockFile)
|
||||||
|
}
|
||||||
|
if len(policy.Services) == 0 || len(policy.Services) > 128 {
|
||||||
|
return errors.New("server policy must allow 1 to 128 services")
|
||||||
|
}
|
||||||
|
for name, service := range policy.Services {
|
||||||
|
if !servicePattern.MatchString(name) {
|
||||||
|
return fmt.Errorf("invalid service name %q", name)
|
||||||
|
}
|
||||||
|
expected := filepath.Join(policy.ConfigRoot, name+".json")
|
||||||
|
if service.Config != expected {
|
||||||
|
return fmt.Errorf("service %s config must be %s", name, expected)
|
||||||
|
}
|
||||||
|
if service.MaxArtifactBytes < 1<<20 || service.MaxArtifactBytes > 512<<20 {
|
||||||
|
return fmt.Errorf("service %s artifact limit is invalid", name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (policy Policy) CheckFiles() ([]CheckedService, error) {
|
||||||
|
if err := policy.CheckDirectories(); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
names := make([]string, 0, len(policy.Services))
|
||||||
|
for name := range policy.Services {
|
||||||
|
names = append(names, name)
|
||||||
|
}
|
||||||
|
sort.Strings(names)
|
||||||
|
checked := make([]CheckedService, 0, len(names))
|
||||||
|
for _, name := range names {
|
||||||
|
service, err := policy.CheckService(name)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
checked = append(checked, service)
|
||||||
|
}
|
||||||
|
return checked, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (policy Policy) CheckDirectories() error {
|
||||||
|
if err := secureDirectory(policy.ConfigRoot, 0); err != nil {
|
||||||
|
return fmt.Errorf("config root: %w", err)
|
||||||
|
}
|
||||||
|
if err := secureDirectory(policy.IncomingRoot, 0o700); err != nil {
|
||||||
|
return fmt.Errorf("incoming root: %w", err)
|
||||||
|
}
|
||||||
|
if err := secureDirectory("/etc/tend/environment", 0o700); err != nil {
|
||||||
|
return fmt.Errorf("environment root: %w", err)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (policy Policy) CheckService(name string) (CheckedService, error) {
|
||||||
|
entry, ok := policy.Services[name]
|
||||||
|
if !ok {
|
||||||
|
return CheckedService{}, errors.New("service is not allowed by server policy")
|
||||||
|
}
|
||||||
|
if err := secureFile(entry.Config, 0); err != nil {
|
||||||
|
return CheckedService{}, fmt.Errorf("service %s config: %w", name, err)
|
||||||
|
}
|
||||||
|
cfg, err := config.Load(entry.Config)
|
||||||
|
if err != nil {
|
||||||
|
return CheckedService{}, fmt.Errorf("service %s config: %w", name, err)
|
||||||
|
}
|
||||||
|
if cfg.Service.Name != name {
|
||||||
|
return CheckedService{}, fmt.Errorf("service %s config identity does not match", name)
|
||||||
|
}
|
||||||
|
if cfg.Deployment.LockFile != policy.SharedLockFile {
|
||||||
|
return CheckedService{}, fmt.Errorf("service %s does not use the host-wide lock", name)
|
||||||
|
}
|
||||||
|
if err = secureFile(cfg.Service.EnvironmentFile, 0o600); err != nil {
|
||||||
|
return CheckedService{}, fmt.Errorf("service %s environment file: %w", name, err)
|
||||||
|
}
|
||||||
|
if cfg.Deployment.Singleton != nil {
|
||||||
|
if err = rejectEnvironmentKey(cfg.Service.EnvironmentFile, cfg.Deployment.Singleton.ListenEnv); err != nil {
|
||||||
|
return CheckedService{}, fmt.Errorf("service %s environment file: %w", name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return CheckedService{Name: name, Config: cfg, Policy: entry}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func CheckConfig(path string, cfg config.Config) error {
|
||||||
|
expected := filepath.Join("/etc/tend/services", cfg.Service.Name+".json")
|
||||||
|
if path != expected {
|
||||||
|
return fmt.Errorf("production config must be %s", expected)
|
||||||
|
}
|
||||||
|
if err := secureDirectory("/etc/tend/services", 0); err != nil {
|
||||||
|
return fmt.Errorf("config root: %w", err)
|
||||||
|
}
|
||||||
|
if err := secureDirectory("/etc/tend/environment", 0o700); err != nil {
|
||||||
|
return fmt.Errorf("environment root: %w", err)
|
||||||
|
}
|
||||||
|
if err := secureFile(path, 0); err != nil {
|
||||||
|
return fmt.Errorf("production config: %w", err)
|
||||||
|
}
|
||||||
|
if cfg.Deployment.LockFile != config.SharedLockFile {
|
||||||
|
return fmt.Errorf("deployment lock must be %s", config.SharedLockFile)
|
||||||
|
}
|
||||||
|
if err := secureFile(cfg.Service.EnvironmentFile, 0o600); err != nil {
|
||||||
|
return fmt.Errorf("environment file: %w", err)
|
||||||
|
}
|
||||||
|
if cfg.Deployment.Singleton != nil {
|
||||||
|
if err := rejectEnvironmentKey(cfg.Service.EnvironmentFile, cfg.Deployment.Singleton.ListenEnv); err != nil {
|
||||||
|
return fmt.Errorf("environment file: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func rejectEnvironmentKey(path, key string) error {
|
||||||
|
file, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
defer file.Close()
|
||||||
|
body, err := io.ReadAll(io.LimitReader(file, 1<<20+1))
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if len(body) > 1<<20 {
|
||||||
|
return errors.New("environment file exceeds 1 MiB")
|
||||||
|
}
|
||||||
|
for _, raw := range bytes.Split(body, []byte{'\n'}) {
|
||||||
|
line := strings.TrimSpace(string(raw))
|
||||||
|
if !strings.HasPrefix(line, key) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
remainder := strings.TrimSpace(strings.TrimPrefix(line, key))
|
||||||
|
if strings.HasPrefix(remainder, "=") {
|
||||||
|
return errors.New("singleton candidate listen key must not be set in the shared environment file")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func secureFile(path string, exactMode os.FileMode) error {
|
||||||
|
if !filepath.IsAbs(path) || filepath.Clean(path) != path || strings.ContainsRune(path, '\x00') {
|
||||||
|
return errors.New("path must be clean and absolute")
|
||||||
|
}
|
||||||
|
if err := rejectSymlinkAncestors(filepath.Dir(path)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return errors.New("must be a regular non-symlink file")
|
||||||
|
}
|
||||||
|
if !rootOwned(info) {
|
||||||
|
return errors.New("must be owned by root")
|
||||||
|
}
|
||||||
|
if exactMode != 0 && info.Mode().Perm() != exactMode {
|
||||||
|
return fmt.Errorf("mode must be %04o", exactMode)
|
||||||
|
}
|
||||||
|
if exactMode == 0 && info.Mode().Perm()&0o022 != 0 {
|
||||||
|
return errors.New("must not be group- or world-writable")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func secureDirectory(path string, exactMode os.FileMode) error {
|
||||||
|
if !filepath.IsAbs(path) || filepath.Clean(path) != path || strings.ContainsRune(path, '\x00') {
|
||||||
|
return errors.New("path must be clean and absolute")
|
||||||
|
}
|
||||||
|
if err := rejectSymlinkAncestors(filepath.Dir(path)); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !info.IsDir() || info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return errors.New("must be a real directory")
|
||||||
|
}
|
||||||
|
if !rootOwned(info) {
|
||||||
|
return errors.New("must be owned by root")
|
||||||
|
}
|
||||||
|
if exactMode != 0 && info.Mode().Perm() != exactMode {
|
||||||
|
return fmt.Errorf("mode must be %04o", exactMode)
|
||||||
|
}
|
||||||
|
if exactMode == 0 && info.Mode().Perm()&0o022 != 0 {
|
||||||
|
return errors.New("must not be group- or world-writable")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func rejectSymlinkAncestors(path string) error {
|
||||||
|
current := string(filepath.Separator)
|
||||||
|
for _, part := range strings.Split(strings.TrimPrefix(filepath.Clean(path), string(filepath.Separator)), string(filepath.Separator)) {
|
||||||
|
if part == "" {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
current = filepath.Join(current, part)
|
||||||
|
info, err := os.Lstat(current)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !info.IsDir() || info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return fmt.Errorf("symlink or non-directory ancestor refused: %s", current)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
package serverpolicy
|
||||||
|
|
||||||
|
import (
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func TestParseAcceptsStrictServiceMap(t *testing.T) {
|
||||||
|
body := `{"schema_version":1,"config_root":"/etc/tend/services","incoming_root":"/var/lib/tend/incoming","shared_lock_file":"/run/lock/tend-deploy.lock","services":{"example-site":{"config":"/etc/tend/services/example-site.json","max_artifact_bytes":1048576}}}`
|
||||||
|
policy, err := Parse(strings.NewReader(body))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if policy.Services["example-site"].MaxArtifactBytes != 1<<20 {
|
||||||
|
t.Fatalf("policy=%+v", policy)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRejectEnvironmentKeyProtectsSingletonCandidateOverride(t *testing.T) {
|
||||||
|
directory := t.TempDir()
|
||||||
|
path := filepath.Join(directory, "service.env")
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
body string
|
||||||
|
wantErr bool
|
||||||
|
}{
|
||||||
|
{name: "shared values only", body: "APP_SECRET=private\n"},
|
||||||
|
{name: "exact listen key", body: "EXAMPLE_LISTEN=127.0.0.1:8092\n", wantErr: true},
|
||||||
|
{name: "spaced listen key", body: " EXAMPLE_LISTEN = 127.0.0.1:8092\n", wantErr: true},
|
||||||
|
{name: "commented listen key", body: "# EXAMPLE_LISTEN=127.0.0.1:8092\n"},
|
||||||
|
{name: "longer key", body: "EXAMPLE_LISTENER=safe\n"},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
if err := os.WriteFile(path, []byte(test.body), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
err := rejectEnvironmentKey(path, "EXAMPLE_LISTEN")
|
||||||
|
if (err != nil) != test.wantErr {
|
||||||
|
t.Fatalf("error=%v wantErr=%v", err, test.wantErr)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestParseRejectsUnknownAndForgedPaths(t *testing.T) {
|
||||||
|
tests := []string{
|
||||||
|
`{"schema_version":1,"config_root":"/etc/tend/services","incoming_root":"/var/lib/tend/incoming","shared_lock_file":"/run/lock/tend-deploy.lock","services":{"example-site":{"config":"/tmp/example.json","max_artifact_bytes":1048576}}}`,
|
||||||
|
`{"schema_version":1,"config_root":"/etc/tend/services","incoming_root":"/var/lib/tend/incoming","shared_lock_file":"/run/lock/tend-deploy.lock","services":{},"surprise":true}`,
|
||||||
|
}
|
||||||
|
for _, body := range tests {
|
||||||
|
if _, err := Parse(strings.NewReader(body)); err == nil {
|
||||||
|
t.Fatalf("accepted %s", body)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -19,10 +19,15 @@ const SchemaVersion = 1
|
|||||||
type Record struct {
|
type Record struct {
|
||||||
SchemaVersion int `json:"schema_version"`
|
SchemaVersion int `json:"schema_version"`
|
||||||
Strategy string `json:"strategy"`
|
Strategy string `json:"strategy"`
|
||||||
|
DesiredRelease string `json:"desired_release,omitempty"`
|
||||||
|
CandidateRelease string `json:"candidate_release,omitempty"`
|
||||||
ActiveSlot string `json:"active_slot"`
|
ActiveSlot string `json:"active_slot"`
|
||||||
ActiveRelease string `json:"active_release"`
|
ActiveRelease string `json:"active_release"`
|
||||||
PreviousSlot string `json:"previous_slot,omitempty"`
|
PreviousSlot string `json:"previous_slot,omitempty"`
|
||||||
PreviousRelease string `json:"previous_release,omitempty"`
|
PreviousRelease string `json:"previous_release,omitempty"`
|
||||||
|
LastAttemptRelease string `json:"last_attempt_release,omitempty"`
|
||||||
|
LastAttemptOutcome string `json:"last_attempt_outcome,omitempty"`
|
||||||
|
LastAttemptAt string `json:"last_attempt_at,omitempty"`
|
||||||
UpdatedAt string `json:"updated_at"`
|
UpdatedAt string `json:"updated_at"`
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -68,6 +73,26 @@ func (r Record) Validate(root, strategy string) error {
|
|||||||
return fmt.Errorf("previous release: %w", err)
|
return fmt.Errorf("previous release: %w", err)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
for label, release := range map[string]string{"desired release": r.DesiredRelease, "candidate release": r.CandidateRelease, "last attempt release": r.LastAttemptRelease} {
|
||||||
|
if release != "" {
|
||||||
|
if err := releaseBelow(root, release); err != nil {
|
||||||
|
return fmt.Errorf("%s: %w", label, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if r.LastAttemptOutcome != "" {
|
||||||
|
switch r.LastAttemptOutcome {
|
||||||
|
case "running", "succeeded", "failed", "rolled_back":
|
||||||
|
default:
|
||||||
|
return errors.New("last attempt outcome is invalid")
|
||||||
|
}
|
||||||
|
if _, err := time.Parse(time.RFC3339, r.LastAttemptAt); err != nil {
|
||||||
|
return errors.New("last attempt timestamp is invalid")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if r.LastAttemptOutcome == "running" && r.CandidateRelease == "" {
|
||||||
|
return errors.New("running attempt requires a candidate release")
|
||||||
|
}
|
||||||
if strategy == "blue_green" && r.PreviousRelease != "" && r.PreviousSlot == r.ActiveSlot {
|
if strategy == "blue_green" && r.PreviousRelease != "" && r.PreviousSlot == r.ActiveSlot {
|
||||||
return errors.New("previous slot must differ from active slot")
|
return errors.New("previous slot must differ from active slot")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -16,7 +16,8 @@ func TestStoreLoadRoundTripAndRejectSymlink(t *testing.T) {
|
|||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
path := filepath.Join(root, "state.json")
|
path := filepath.Join(root, "state.json")
|
||||||
record := Record{SchemaVersion: 1, Strategy: "singleton_candidate", ActiveSlot: "singleton", ActiveRelease: release, UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
|
at := time.Unix(1, 0).UTC().Format(time.RFC3339)
|
||||||
|
record := Record{SchemaVersion: 1, Strategy: "singleton_candidate", DesiredRelease: release, ActiveSlot: "singleton", ActiveRelease: release, LastAttemptRelease: release, LastAttemptOutcome: "succeeded", LastAttemptAt: at, UpdatedAt: at}
|
||||||
if err := Store(path, root, record); err != nil {
|
if err := Store(path, root, record); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
@@ -24,8 +25,8 @@ func TestStoreLoadRoundTripAndRejectSymlink(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
}
|
}
|
||||||
if loaded.ActiveRelease != release {
|
if loaded.ActiveRelease != release || loaded.DesiredRelease != release || loaded.LastAttemptOutcome != "succeeded" {
|
||||||
t.Fatalf("release=%q", loaded.ActiveRelease)
|
t.Fatalf("state=%+v", loaded)
|
||||||
}
|
}
|
||||||
if err := os.Remove(path); err != nil {
|
if err := os.Remove(path); err != nil {
|
||||||
t.Fatal(err)
|
t.Fatal(err)
|
||||||
@@ -37,6 +38,15 @@ func TestStoreLoadRoundTripAndRejectSymlink(t *testing.T) {
|
|||||||
t.Fatal("expected symlink refusal")
|
t.Fatal("expected symlink refusal")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestRecordRequiresCandidateForRunningAttempt(t *testing.T) {
|
||||||
|
root := filepath.Join(t.TempDir(), "service")
|
||||||
|
release := filepath.Join(root, "releases", "sha256-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
|
||||||
|
record := Record{SchemaVersion: 1, Strategy: "singleton_candidate", DesiredRelease: release, ActiveSlot: "singleton", ActiveRelease: release, LastAttemptRelease: release, LastAttemptOutcome: "running", LastAttemptAt: time.Unix(1, 0).UTC().Format(time.RFC3339), UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
|
||||||
|
if err := record.Validate(root, "singleton_candidate"); err == nil {
|
||||||
|
t.Fatal("expected missing candidate rejection")
|
||||||
|
}
|
||||||
|
}
|
||||||
func TestRecordRejectsReleaseOutsideRoot(t *testing.T) {
|
func TestRecordRejectsReleaseOutsideRoot(t *testing.T) {
|
||||||
record := Record{SchemaVersion: 1, Strategy: "singleton_candidate", ActiveSlot: "singleton", ActiveRelease: "/tmp/other/release", UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
|
record := Record{SchemaVersion: 1, Strategy: "singleton_candidate", ActiveSlot: "singleton", ActiveRelease: "/tmp/other/release", UpdatedAt: time.Unix(1, 0).UTC().Format(time.RFC3339)}
|
||||||
if err := record.Validate("/opt/example", "singleton_candidate"); err == nil {
|
if err := record.Validate("/opt/example", "singleton_candidate"); err == nil {
|
||||||
|
|||||||
@@ -0,0 +1,123 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
// Package transport implements Tend's bounded, versioned deployment stream.
|
||||||
|
package transport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/binary"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
const (
|
||||||
|
Protocol = "tend-receive-v1"
|
||||||
|
MaxArtifactBytes = 512 << 20
|
||||||
|
maxHeaderBytes = 64 << 10
|
||||||
|
maxArtifactName = 128
|
||||||
|
)
|
||||||
|
|
||||||
|
var (
|
||||||
|
servicePattern = regexp.MustCompile(`^[a-z][a-z0-9-]{1,62}$`)
|
||||||
|
artifactPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`)
|
||||||
|
)
|
||||||
|
|
||||||
|
type Header struct {
|
||||||
|
Protocol string `json:"protocol"`
|
||||||
|
Service string `json:"service"`
|
||||||
|
ArtifactName string `json:"artifact_name"`
|
||||||
|
Size int64 `json:"size"`
|
||||||
|
SHA256 string `json:"sha256"`
|
||||||
|
ApprovedSHA256 string `json:"approved_sha256"`
|
||||||
|
Activate bool `json:"activate"`
|
||||||
|
}
|
||||||
|
|
||||||
|
func (h Header) Validate(maxBytes int64) error {
|
||||||
|
if h.Protocol != Protocol {
|
||||||
|
return errors.New("unsupported receive protocol")
|
||||||
|
}
|
||||||
|
if !servicePattern.MatchString(h.Service) {
|
||||||
|
return errors.New("invalid service name")
|
||||||
|
}
|
||||||
|
if len(h.ArtifactName) > maxArtifactName || !artifactPattern.MatchString(h.ArtifactName) {
|
||||||
|
return errors.New("invalid artifact name")
|
||||||
|
}
|
||||||
|
if h.Size <= 0 || h.Size > maxBytes {
|
||||||
|
return errors.New("artifact size exceeds policy")
|
||||||
|
}
|
||||||
|
if h.SHA256 != h.ApprovedSHA256 || len(h.SHA256) != 64 || strings.ToLower(h.SHA256) != h.SHA256 {
|
||||||
|
return errors.New("artifact digest was not explicitly approved")
|
||||||
|
}
|
||||||
|
if _, err := hex.DecodeString(h.SHA256); err != nil {
|
||||||
|
return errors.New("artifact digest is not hexadecimal")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func Prefix(header Header) ([]byte, error) {
|
||||||
|
if err := header.Validate(MaxArtifactBytes); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
body, err := json.Marshal(header)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if len(body) > maxHeaderBytes {
|
||||||
|
return nil, errors.New("receive header exceeds limit")
|
||||||
|
}
|
||||||
|
prefix := make([]byte, 4+len(body))
|
||||||
|
binary.BigEndian.PutUint32(prefix[:4], uint32(len(body)))
|
||||||
|
copy(prefix[4:], body)
|
||||||
|
return prefix, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func ReadHeader(reader *bufio.Reader) (Header, error) {
|
||||||
|
var size [4]byte
|
||||||
|
if _, err := io.ReadFull(reader, size[:]); err != nil {
|
||||||
|
return Header{}, fmt.Errorf("read receive header length: %w", err)
|
||||||
|
}
|
||||||
|
length := binary.BigEndian.Uint32(size[:])
|
||||||
|
if length == 0 || length > maxHeaderBytes {
|
||||||
|
return Header{}, errors.New("receive header length is invalid")
|
||||||
|
}
|
||||||
|
body := make([]byte, length)
|
||||||
|
if _, err := io.ReadFull(reader, body); err != nil {
|
||||||
|
return Header{}, fmt.Errorf("read receive header: %w", err)
|
||||||
|
}
|
||||||
|
decoder := json.NewDecoder(bytes.NewReader(body))
|
||||||
|
decoder.DisallowUnknownFields()
|
||||||
|
var header Header
|
||||||
|
if err := decoder.Decode(&header); err != nil {
|
||||||
|
return Header{}, fmt.Errorf("decode receive header: %w", err)
|
||||||
|
}
|
||||||
|
var trailing any
|
||||||
|
if err := decoder.Decode(&trailing); !errors.Is(err, io.EOF) {
|
||||||
|
return Header{}, errors.New("receive header contains trailing data")
|
||||||
|
}
|
||||||
|
return header, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func CopyArtifact(destination io.Writer, reader *bufio.Reader, header Header, maxBytes int64) error {
|
||||||
|
if err := header.Validate(maxBytes); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
hash := sha256.New()
|
||||||
|
written, err := io.CopyN(io.MultiWriter(destination, hash), reader, header.Size)
|
||||||
|
if err != nil || written != header.Size {
|
||||||
|
return errors.New("artifact stream ended before declared size")
|
||||||
|
}
|
||||||
|
if _, err = reader.ReadByte(); !errors.Is(err, io.EOF) {
|
||||||
|
return errors.New("artifact stream contains trailing bytes")
|
||||||
|
}
|
||||||
|
if hex.EncodeToString(hash.Sum(nil)) != header.SHA256 {
|
||||||
|
return errors.New("artifact stream digest does not match")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
package transport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"strings"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
func validFrame(t *testing.T, artifact string) ([]byte, Header) {
|
||||||
|
t.Helper()
|
||||||
|
hash := sha256.Sum256([]byte(artifact))
|
||||||
|
digest := hex.EncodeToString(hash[:])
|
||||||
|
header := Header{Protocol: Protocol, Service: "example-site", ArtifactName: "example.tar.gz", Size: int64(len(artifact)), SHA256: digest, ApprovedSHA256: digest, Activate: true}
|
||||||
|
prefix, err := Prefix(header)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
return append(prefix, artifact...), header
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProtocolRoundTrip(t *testing.T) {
|
||||||
|
frame, expected := validFrame(t, "artifact")
|
||||||
|
reader := bufio.NewReader(bytes.NewReader(frame))
|
||||||
|
header, err := ReadHeader(reader)
|
||||||
|
if err != nil || header != expected {
|
||||||
|
t.Fatalf("header=%+v err=%v", header, err)
|
||||||
|
}
|
||||||
|
var artifact bytes.Buffer
|
||||||
|
if err = CopyArtifact(&artifact, reader, header, 1<<20); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if artifact.String() != "artifact" {
|
||||||
|
t.Fatalf("artifact=%q", artifact.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestProtocolRejectsTrailingAndForgedInputs(t *testing.T) {
|
||||||
|
frame, header := validFrame(t, "artifact")
|
||||||
|
reader := bufio.NewReader(bytes.NewReader(append(frame, 'x')))
|
||||||
|
read, _ := ReadHeader(reader)
|
||||||
|
if err := CopyArtifact(&bytes.Buffer{}, reader, read, 1<<20); err == nil {
|
||||||
|
t.Fatal("accepted trailing bytes")
|
||||||
|
}
|
||||||
|
header.Service = "../../root"
|
||||||
|
if _, err := Prefix(header); err == nil {
|
||||||
|
t.Fatal("accepted forged service")
|
||||||
|
}
|
||||||
|
header.Service = "example-site"
|
||||||
|
header.ApprovedSHA256 = strings.Repeat("0", 64)
|
||||||
|
if _, err := Prefix(header); err == nil {
|
||||||
|
t.Fatal("accepted unapproved digest")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func FuzzProtocolFraming(f *testing.F) {
|
||||||
|
hash := sha256.Sum256([]byte("artifact"))
|
||||||
|
digest := hex.EncodeToString(hash[:])
|
||||||
|
prefix, err := Prefix(Header{Protocol: Protocol, Service: "example-site", ArtifactName: "example.tar.gz", Size: 8, SHA256: digest, ApprovedSHA256: digest})
|
||||||
|
if err != nil {
|
||||||
|
f.Fatal(err)
|
||||||
|
}
|
||||||
|
frame := append(prefix, []byte("artifact")...)
|
||||||
|
f.Add(frame)
|
||||||
|
f.Add([]byte{0, 0, 0, 0})
|
||||||
|
f.Fuzz(func(t *testing.T, input []byte) {
|
||||||
|
if len(input) > 2<<20 {
|
||||||
|
t.Skip()
|
||||||
|
}
|
||||||
|
reader := bufio.NewReader(bytes.NewReader(input))
|
||||||
|
header, err := ReadHeader(reader)
|
||||||
|
if err != nil {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
_ = CopyArtifact(&bytes.Buffer{}, reader, header, 1<<20)
|
||||||
|
})
|
||||||
|
}
|
||||||
@@ -0,0 +1,153 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
package transport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"encoding/json"
|
||||||
|
"errors"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"os/exec"
|
||||||
|
"path/filepath"
|
||||||
|
"regexp"
|
||||||
|
"strconv"
|
||||||
|
"strings"
|
||||||
|
)
|
||||||
|
|
||||||
|
var targetPattern = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9_.-]*@[A-Za-z0-9][A-Za-z0-9.-]*$`)
|
||||||
|
|
||||||
|
type PushOptions struct {
|
||||||
|
Target string
|
||||||
|
Port int
|
||||||
|
KnownHosts string
|
||||||
|
Identity string
|
||||||
|
Service string
|
||||||
|
Artifact string
|
||||||
|
SHA256 string
|
||||||
|
ApprovedSHA256 string
|
||||||
|
Activate bool
|
||||||
|
}
|
||||||
|
|
||||||
|
type SSHRunner interface {
|
||||||
|
Run(context.Context, string, []string, io.Reader) ([]byte, error)
|
||||||
|
}
|
||||||
|
|
||||||
|
type ExecSSHRunner struct{}
|
||||||
|
|
||||||
|
func (ExecSSHRunner) Run(ctx context.Context, name string, args []string, stdin io.Reader) ([]byte, error) {
|
||||||
|
command := exec.CommandContext(ctx, name, args...)
|
||||||
|
command.Stdin = stdin
|
||||||
|
var output, diagnostic boundedBuffer
|
||||||
|
command.Stdout = &output
|
||||||
|
command.Stderr = &diagnostic
|
||||||
|
err := command.Run()
|
||||||
|
if err != nil {
|
||||||
|
return output.Bytes(), fmt.Errorf("ssh failed: %w: %s", err, strings.TrimSpace(diagnostic.String()))
|
||||||
|
}
|
||||||
|
return output.Bytes(), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
type boundedBuffer struct{ bytes.Buffer }
|
||||||
|
|
||||||
|
func (b *boundedBuffer) Write(value []byte) (int, error) {
|
||||||
|
written := len(value)
|
||||||
|
remaining := (1 << 20) - b.Len()
|
||||||
|
if remaining > 0 {
|
||||||
|
if len(value) > remaining {
|
||||||
|
value = value[:remaining]
|
||||||
|
}
|
||||||
|
_, _ = b.Buffer.Write(value)
|
||||||
|
}
|
||||||
|
return written, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func Push(ctx context.Context, runner SSHRunner, options PushOptions) (json.RawMessage, error) {
|
||||||
|
if !targetPattern.MatchString(options.Target) || strings.HasPrefix(options.Target, "-") {
|
||||||
|
return nil, errors.New("target must be user@host without shell syntax")
|
||||||
|
}
|
||||||
|
if options.Port < 1 || options.Port > 65535 {
|
||||||
|
return nil, errors.New("SSH port is invalid")
|
||||||
|
}
|
||||||
|
if err := safeClientFile(options.KnownHosts, false); err != nil {
|
||||||
|
return nil, fmt.Errorf("known-hosts file: %w", err)
|
||||||
|
}
|
||||||
|
if options.Identity != "" {
|
||||||
|
if err := safeClientFile(options.Identity, true); err != nil {
|
||||||
|
return nil, fmt.Errorf("identity file: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !filepath.IsAbs(options.Artifact) || filepath.Clean(options.Artifact) != options.Artifact || strings.ContainsAny(options.Artifact, "\x00\r\n\t") {
|
||||||
|
return nil, errors.New("artifact path must be clean and absolute")
|
||||||
|
}
|
||||||
|
artifactInfo, err := os.Lstat(options.Artifact)
|
||||||
|
if err != nil || !artifactInfo.Mode().IsRegular() || artifactInfo.Mode()&os.ModeSymlink != 0 || artifactInfo.Size() <= 0 || artifactInfo.Size() > MaxArtifactBytes {
|
||||||
|
return nil, errors.New("artifact must be a bounded regular non-symlink file")
|
||||||
|
}
|
||||||
|
artifact, err := os.Open(options.Artifact)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer artifact.Close()
|
||||||
|
info, err := artifact.Stat()
|
||||||
|
if err != nil || !info.Mode().IsRegular() || !os.SameFile(artifactInfo, info) {
|
||||||
|
return nil, errors.New("artifact must be a bounded regular file")
|
||||||
|
}
|
||||||
|
hash := sha256.New()
|
||||||
|
if _, err = io.Copy(hash, artifact); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
actual := hex.EncodeToString(hash.Sum(nil))
|
||||||
|
if actual != options.SHA256 || options.SHA256 != options.ApprovedSHA256 {
|
||||||
|
return nil, errors.New("artifact digest was not explicitly approved")
|
||||||
|
}
|
||||||
|
if _, err = artifact.Seek(0, io.SeekStart); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
header := Header{Protocol: Protocol, Service: options.Service, ArtifactName: filepath.Base(options.Artifact), Size: info.Size(), SHA256: options.SHA256, ApprovedSHA256: options.ApprovedSHA256, Activate: options.Activate}
|
||||||
|
prefix, err := Prefix(header)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
args := []string{"-F", os.DevNull, "-T", "-p", strconv.Itoa(options.Port), "-o", "BatchMode=yes", "-o", "ClearAllForwardings=yes", "-o", "ExitOnForwardFailure=yes", "-o", "ForwardAgent=no", "-o", "IdentitiesOnly=yes", "-o", "LogLevel=ERROR", "-o", "PermitLocalCommand=no", "-o", "ProxyCommand=none", "-o", "RequestTTY=no", "-o", "StrictHostKeyChecking=yes", "-o", "UserKnownHostsFile=" + options.KnownHosts}
|
||||||
|
if options.Identity != "" {
|
||||||
|
args = append(args, "-i", options.Identity)
|
||||||
|
}
|
||||||
|
args = append(args, options.Target, Protocol)
|
||||||
|
output, err := runner.Run(ctx, "ssh", args, io.MultiReader(bytes.NewReader(prefix), artifact))
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
if !json.Valid(output) {
|
||||||
|
return nil, errors.New("receiver returned invalid JSON")
|
||||||
|
}
|
||||||
|
return json.RawMessage(output), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func safeClientFile(path string, private bool) error {
|
||||||
|
if !filepath.IsAbs(path) || filepath.Clean(path) != path || strings.ContainsAny(path, "\x00\r\n\t") {
|
||||||
|
return errors.New("path must be clean and absolute")
|
||||||
|
}
|
||||||
|
parent, err := os.Lstat(filepath.Dir(path))
|
||||||
|
if err != nil || !parent.IsDir() || parent.Mode()&os.ModeSymlink != 0 || parent.Mode().Perm()&0o022 != 0 {
|
||||||
|
return errors.New("parent must be a real directory not writable by group or others")
|
||||||
|
}
|
||||||
|
info, err := os.Lstat(path)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !info.Mode().IsRegular() || info.Mode()&os.ModeSymlink != 0 {
|
||||||
|
return errors.New("must be a regular non-symlink file")
|
||||||
|
}
|
||||||
|
if info.Mode().Perm()&0o022 != 0 {
|
||||||
|
return errors.New("must not be group- or world-writable")
|
||||||
|
}
|
||||||
|
if private && info.Mode().Perm()&0o077 != 0 {
|
||||||
|
return errors.New("must not be accessible by group or others")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
@@ -0,0 +1,81 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
package transport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"crypto/sha256"
|
||||||
|
"encoding/hex"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
"slices"
|
||||||
|
"testing"
|
||||||
|
)
|
||||||
|
|
||||||
|
type captureRunner struct {
|
||||||
|
name string
|
||||||
|
args []string
|
||||||
|
input []byte
|
||||||
|
calls int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (runner *captureRunner) Run(_ context.Context, name string, args []string, input io.Reader) ([]byte, error) {
|
||||||
|
runner.calls++
|
||||||
|
runner.name = name
|
||||||
|
runner.args = append([]string(nil), args...)
|
||||||
|
runner.input, _ = io.ReadAll(input)
|
||||||
|
return []byte(`{"validated":true,"mutation":"activated"}`), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPushUsesPinnedSSHAndExactFrame(t *testing.T) {
|
||||||
|
dir := t.TempDir()
|
||||||
|
knownHosts := filepath.Join(dir, "known_hosts")
|
||||||
|
identity := filepath.Join(dir, "identity")
|
||||||
|
artifact := filepath.Join(dir, "release.tar.gz")
|
||||||
|
if err := os.WriteFile(knownHosts, []byte("host key\n"), 0o644); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(identity, []byte("private\n"), 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
content := []byte("artifact")
|
||||||
|
if err := os.WriteFile(artifact, content, 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
hash := sha256.Sum256(content)
|
||||||
|
digest := hex.EncodeToString(hash[:])
|
||||||
|
runner := &captureRunner{}
|
||||||
|
result, err := Push(context.Background(), runner, PushOptions{Target: "tend-deploy@example.test", Port: 2222, KnownHosts: knownHosts, Identity: identity, Service: "example-site", Artifact: artifact, SHA256: digest, ApprovedSHA256: digest, Activate: true})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if !bytes.Contains(result, []byte(`"activated"`)) || runner.calls != 1 || runner.name != "ssh" {
|
||||||
|
t.Fatalf("result=%s calls=%d name=%q", result, runner.calls, runner.name)
|
||||||
|
}
|
||||||
|
if !slices.Contains(runner.args, "ProxyCommand=none") || !slices.Contains(runner.args, "StrictHostKeyChecking=yes") || runner.args[len(runner.args)-1] != Protocol {
|
||||||
|
t.Fatalf("args=%#v", runner.args)
|
||||||
|
}
|
||||||
|
reader := bufio.NewReader(bytes.NewReader(runner.input))
|
||||||
|
header, err := ReadHeader(reader)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
var copied bytes.Buffer
|
||||||
|
if err = CopyArtifact(&copied, reader, header, 1<<20); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
if copied.String() != string(content) || header.Service != "example-site" || !header.Activate {
|
||||||
|
t.Fatalf("header=%+v body=%q", header, copied.String())
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPushRejectsShellTargetBeforeExecution(t *testing.T) {
|
||||||
|
runner := &captureRunner{}
|
||||||
|
_, err := Push(context.Background(), runner, PushOptions{Target: "root@example.test;touch", Port: 22})
|
||||||
|
if err == nil || runner.calls != 0 {
|
||||||
|
t.Fatalf("err=%v calls=%d", err, runner.calls)
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
// SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
|
||||||
|
package transport
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bufio"
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"os"
|
||||||
|
|
||||||
|
"gamertan.com/tend/internal/deploy"
|
||||||
|
"gamertan.com/tend/internal/serverpolicy"
|
||||||
|
)
|
||||||
|
|
||||||
|
func Receive(ctx context.Context, input io.Reader, policy serverpolicy.Policy, manager deploy.Manager) (deploy.Report, error) {
|
||||||
|
if err := policy.CheckDirectories(); err != nil {
|
||||||
|
return deploy.Report{}, err
|
||||||
|
}
|
||||||
|
reader := bufio.NewReaderSize(input, maxHeaderBytes+4)
|
||||||
|
header, err := ReadHeader(reader)
|
||||||
|
if err != nil {
|
||||||
|
return deploy.Report{}, err
|
||||||
|
}
|
||||||
|
service, err := policy.CheckService(header.Service)
|
||||||
|
if err != nil {
|
||||||
|
return deploy.Report{}, err
|
||||||
|
}
|
||||||
|
if err = header.Validate(service.Policy.MaxArtifactBytes); err != nil {
|
||||||
|
return deploy.Report{}, err
|
||||||
|
}
|
||||||
|
file, err := os.CreateTemp(policy.IncomingRoot, ".tend-receive-"+header.Service+"-")
|
||||||
|
if err != nil {
|
||||||
|
return deploy.Report{}, err
|
||||||
|
}
|
||||||
|
path := file.Name()
|
||||||
|
defer os.Remove(path)
|
||||||
|
if err = file.Chmod(0o600); err == nil {
|
||||||
|
err = CopyArtifact(file, reader, header, service.Policy.MaxArtifactBytes)
|
||||||
|
}
|
||||||
|
if closeErr := file.Close(); err == nil {
|
||||||
|
err = closeErr
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return deploy.Report{}, fmt.Errorf("receive artifact: %w", err)
|
||||||
|
}
|
||||||
|
return manager.Deploy(ctx, service.Config, deploy.Request{Artifact: path, SHA256: header.SHA256, ApprovedSHA256: header.ApprovedSHA256, Activate: header.Activate})
|
||||||
|
}
|
||||||
+10
-5
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"schema_version": 1,
|
"schema_version": 2,
|
||||||
"service": { "name": "tend", "allowed_host": "localhost" },
|
"service": { "name": "tend", "allowed_host": "localhost", "environment_file": "/etc/tend/environment/tend-release-test.env" },
|
||||||
"build": {
|
"build": {
|
||||||
"package": "./cmd/tend",
|
"package": "./cmd/tend",
|
||||||
"binary": "tend",
|
"binary": "tend",
|
||||||
@@ -12,20 +12,25 @@
|
|||||||
"deployment": {
|
"deployment": {
|
||||||
"strategy": "singleton_candidate",
|
"strategy": "singleton_candidate",
|
||||||
"root": "/opt/tend-release-test",
|
"root": "/opt/tend-release-test",
|
||||||
"lock_file": "/run/lock/tend-release-test.lock",
|
"lock_file": "/run/lock/tend-deploy.lock",
|
||||||
"state_file": "/opt/tend-release-test/tend-state.json",
|
"state_file": "/opt/tend-release-test/tend-state.json",
|
||||||
|
"event_log": "/opt/tend-release-test/deployment-events.jsonl",
|
||||||
"health_path": "/healthz",
|
"health_path": "/healthz",
|
||||||
"readiness_path": "/readyz",
|
"readiness_path": "/readyz",
|
||||||
"candidate_timeout_seconds": 5,
|
"candidate_timeout_seconds": 5,
|
||||||
|
"activation_window_seconds": 5,
|
||||||
"smoke": [{ "path": "/", "contains": "Tend" }],
|
"smoke": [{ "path": "/", "contains": "Tend" }],
|
||||||
|
"public_smoke": [{ "url": "https://example.test/", "contains": "Tend" }],
|
||||||
"singleton": {
|
"singleton": {
|
||||||
"unit": "tend-release-test.service",
|
"unit": "tend-release-test.service",
|
||||||
"address": "127.0.0.1:19090",
|
"address": "127.0.0.1:19090",
|
||||||
"candidate_address": "127.0.0.1:19091",
|
"candidate_address": "127.0.0.1:19091",
|
||||||
"listen_env": "TEND_RELEASE_TEST_LISTEN",
|
"listen_env": "TEND_RELEASE_TEST_LISTEN",
|
||||||
"environment": {},
|
|
||||||
"current_link": "/opt/tend-release-test/current",
|
"current_link": "/opt/tend-release-test/current",
|
||||||
"previous_link": "/opt/tend-release-test/previous"
|
"previous_link": "/opt/tend-release-test/previous",
|
||||||
|
"caddy_config": "/etc/caddy/Caddyfile",
|
||||||
|
"caddy_handler": "/etc/caddy/tend-release-test-handler.caddy",
|
||||||
|
"caddy_handler_template": "/etc/tend/caddy/tend-release-test.template"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
Executable
+30
@@ -0,0 +1,30 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# SPDX-License-Identifier: AGPL-3.0-only
|
||||||
|
set -euo pipefail
|
||||||
|
root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
|
||||||
|
allow=$root/scripts/public-snapshot.allow
|
||||||
|
LC_ALL=C sort -c "$allow"
|
||||||
|
[[ $(LC_ALL=C sort "$allow" | uniq -d | wc -l) -eq 0 ]]
|
||||||
|
mapfile -t files <"$allow"
|
||||||
|
[[ ${#files[@]} -gt 0 ]]
|
||||||
|
for file in "${files[@]}"; do
|
||||||
|
[[ -n $file && $file != /* && $file != *..* && $file != .gitea/* && $file != .github/* ]]
|
||||||
|
git -C "$root" cat-file -e "HEAD:$file"
|
||||||
|
done
|
||||||
|
work=$(mktemp -d)
|
||||||
|
trap 'rm -rf -- "$work"' EXIT
|
||||||
|
mkdir -m 0700 "$work/tree"
|
||||||
|
git -C "$root" archive HEAD -- "${files[@]}" | tar -xf - -C "$work/tree"
|
||||||
|
test ! -e "$work/tree/.git"
|
||||||
|
test ! -e "$work/tree/.gitea"
|
||||||
|
test ! -e "$work/tree/.github"
|
||||||
|
private_pattern='/home/[[:alnum:]_.-]+/|BEGIN (RSA|OPENSSH|EC) PRIVATE KEY|gitea[-_]api[[:alnum:]_.-]*token'
|
||||||
|
if (cd "$work/tree" && rg -n --hidden --glob '!scripts/export-public.sh' --glob '!scripts/check-public-tree.sh' "$private_pattern" .); then
|
||||||
|
echo "private material found in public tree" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
(cd "$work/tree" && ./scripts/check-licenses.sh)
|
||||||
|
(cd "$work/tree" && GOWORK=off go test -count=1 ./...)
|
||||||
|
(cd "$work/tree" && GOWORK=off go vet ./...)
|
||||||
|
(cd "$work/tree" && GOWORK=off CGO_ENABLED=0 go build -buildvcs=false -mod=readonly -trimpath -o "$work/tend" ./cmd/tend)
|
||||||
|
echo "public tree compiles independently"
|
||||||
@@ -44,7 +44,8 @@ done < <(find "$stage" -type f -print | LC_ALL=C sort)
|
|||||||
epoch=$(git -C "$root" show -s --format=%ct "$commit")
|
epoch=$(git -C "$root" show -s --format=%ct "$commit")
|
||||||
printf '{"schema_version":1,"source_commit":"%s","source_tree":"%s","source_date_epoch":%s,"file_count":%d}\n' "$commit" "$tree" "$epoch" "${#files[@]}" >"$stage/PUBLIC-SNAPSHOT.json"
|
printf '{"schema_version":1,"source_commit":"%s","source_tree":"%s","source_date_epoch":%s,"file_count":%d}\n' "$commit" "$tree" "$epoch" "${#files[@]}" >"$stage/PUBLIC-SNAPSHOT.json"
|
||||||
(cd "$stage" && sha256sum PUBLIC-SNAPSHOT.json >PUBLIC-SNAPSHOT.sha256)
|
(cd "$stage" && sha256sum PUBLIC-SNAPSHOT.json >PUBLIC-SNAPSHOT.sha256)
|
||||||
if (cd "$stage" && rg -n --hidden --glob '!.git/**' --glob '!PUBLIC-SNAPSHOT.json' --glob '!scripts/export-public.sh' '/home/cole|BEGIN (RSA|OPENSSH|EC) PRIVATE KEY|gitea-api\.token' .); then
|
private_pattern='/home/[[:alnum:]_.-]+/|BEGIN (RSA|OPENSSH|EC) PRIVATE KEY|gitea[-_]api[[:alnum:]_.-]*token'
|
||||||
|
if (cd "$stage" && rg -n --hidden --glob '!.git/**' --glob '!PUBLIC-SNAPSHOT.json' --glob '!scripts/export-public.sh' "$private_pattern" .); then
|
||||||
echo "private material found" >&2
|
echo "private material found" >&2
|
||||||
exit 1
|
exit 1
|
||||||
fi
|
fi
|
||||||
|
|||||||
@@ -8,13 +8,30 @@ SECURITY.md
|
|||||||
cmd/tend/main.go
|
cmd/tend/main.go
|
||||||
docs/ARCHITECTURE.md
|
docs/ARCHITECTURE.md
|
||||||
docs/DOGFOOD_EVIDENCE.md
|
docs/DOGFOOD_EVIDENCE.md
|
||||||
|
docs/DOGFOOD_FRICTION.md
|
||||||
docs/PUBLIC_SNAPSHOT.md
|
docs/PUBLIC_SNAPSHOT.md
|
||||||
|
docs/SCHEMA_V2_MIGRATION.md
|
||||||
docs/THREAT_MODEL.md
|
docs/THREAT_MODEL.md
|
||||||
|
docs/WALKTHROUGH.md
|
||||||
examples/LICENSE
|
examples/LICENSE
|
||||||
examples/README.md
|
examples/README.md
|
||||||
examples/blue-green/caddy-handler.template
|
examples/blue-green/caddy-handler.template
|
||||||
examples/blue-green/example-site@.service
|
examples/blue-green/example-site@.service
|
||||||
examples/blue-green/tend.json
|
examples/blue-green/tend.json
|
||||||
|
examples/local/.env.example
|
||||||
|
examples/server/authorized_keys.example
|
||||||
|
examples/server/caddy/docs-site.template
|
||||||
|
examples/server/caddy/example-site.template
|
||||||
|
examples/server/environment/docs-site.env.example
|
||||||
|
examples/server/environment/example-site.env.example
|
||||||
|
examples/server/example-singleton.service
|
||||||
|
examples/server/receive-policy.json
|
||||||
|
examples/server/services/docs-site.json
|
||||||
|
examples/server/services/example-site.json
|
||||||
|
examples/server/slots/example-site-blue.env
|
||||||
|
examples/server/slots/example-site-green.env
|
||||||
|
examples/server/tend-receive.sudoers
|
||||||
|
examples/singleton/caddy-handler.template
|
||||||
examples/singleton/tend.json
|
examples/singleton/tend.json
|
||||||
go.mod
|
go.mod
|
||||||
internal/config/config.go
|
internal/config/config.go
|
||||||
@@ -22,23 +39,37 @@ internal/config/config_test.go
|
|||||||
internal/deploy/deploy.go
|
internal/deploy/deploy.go
|
||||||
internal/deploy/deploy_test.go
|
internal/deploy/deploy_test.go
|
||||||
internal/deploy/lock_linux.go
|
internal/deploy/lock_linux.go
|
||||||
|
internal/deploy/lock_linux_test.go
|
||||||
internal/deploy/lock_other.go
|
internal/deploy/lock_other.go
|
||||||
internal/deploy/operator.go
|
internal/deploy/operator.go
|
||||||
internal/deploy/operator_test.go
|
internal/deploy/operator_test.go
|
||||||
internal/deploy/ownership_linux.go
|
internal/deploy/ownership_linux.go
|
||||||
internal/deploy/ownership_other.go
|
internal/deploy/ownership_other.go
|
||||||
internal/deploy/release.go
|
internal/deploy/release.go
|
||||||
|
internal/deploy/release_mode_linux_test.go
|
||||||
internal/deploy/release_test.go
|
internal/deploy/release_test.go
|
||||||
|
internal/eventlog/eventlog.go
|
||||||
|
internal/eventlog/eventlog_test.go
|
||||||
internal/packager/packager.go
|
internal/packager/packager.go
|
||||||
internal/packager/packager_test.go
|
internal/packager/packager_test.go
|
||||||
internal/process/run.go
|
internal/process/run.go
|
||||||
internal/provenance/git.go
|
internal/provenance/git.go
|
||||||
|
internal/serverpolicy/ownership_linux.go
|
||||||
|
internal/serverpolicy/ownership_other.go
|
||||||
|
internal/serverpolicy/policy.go
|
||||||
|
internal/serverpolicy/policy_test.go
|
||||||
internal/state/state.go
|
internal/state/state.go
|
||||||
internal/state/state_test.go
|
internal/state/state_test.go
|
||||||
|
internal/transport/protocol.go
|
||||||
|
internal/transport/protocol_test.go
|
||||||
|
internal/transport/push.go
|
||||||
|
internal/transport/push_test.go
|
||||||
|
internal/transport/receive.go
|
||||||
internal/version/version.go
|
internal/version/version.go
|
||||||
internal/version/version_test.go
|
internal/version/version_test.go
|
||||||
release/tend.json
|
release/tend.json
|
||||||
scripts/check-licenses.sh
|
scripts/check-licenses.sh
|
||||||
|
scripts/check-public-tree.sh
|
||||||
scripts/export-public.sh
|
scripts/export-public.sh
|
||||||
scripts/public-snapshot.allow
|
scripts/public-snapshot.allow
|
||||||
scripts/test-public-snapshot.sh
|
scripts/test-public-snapshot.sh
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ set -euo pipefail
|
|||||||
root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
|
root=$(CDPATH= cd -- "$(dirname -- "$0")/.." && pwd)
|
||||||
cd "$root"
|
cd "$root"
|
||||||
./scripts/check-licenses.sh
|
./scripts/check-licenses.sh
|
||||||
|
./scripts/check-public-tree.sh
|
||||||
go test -count=1 ./...
|
go test -count=1 ./...
|
||||||
go test -race -count=1 ./...
|
go test -race -count=1 ./...
|
||||||
go vet ./...
|
go vet ./...
|
||||||
@@ -14,6 +15,11 @@ GOWORK=off CGO_ENABLED=0 go build -mod=readonly -trimpath -o "$work/tend-2" ./cm
|
|||||||
cmp "$work/tend-1" "$work/tend-2"
|
cmp "$work/tend-1" "$work/tend-2"
|
||||||
"$work/tend-1" check --config "$root/examples/blue-green/tend.json" >/dev/null
|
"$work/tend-1" check --config "$root/examples/blue-green/tend.json" >/dev/null
|
||||||
"$work/tend-1" check --config "$root/examples/singleton/tend.json" >/dev/null
|
"$work/tend-1" check --config "$root/examples/singleton/tend.json" >/dev/null
|
||||||
|
"$work/tend-1" check --config "$root/examples/server/services/example-site.json" >/dev/null
|
||||||
|
"$work/tend-1" check --config "$root/examples/server/services/docs-site.json" >/dev/null
|
||||||
|
"$work/tend-1" check --config "$root/release/tend.json" >/dev/null
|
||||||
|
grep -Fqx 'Defaults:tend-deploy env_keep += "SSH_ORIGINAL_COMMAND"' "$root/examples/server/tend-receive.sudoers"
|
||||||
|
grep -Fqx 'tend-deploy ALL=(root) NOPASSWD: /usr/local/bin/tend receive --policy /etc/tend/receive-policy.json' "$root/examples/server/tend-receive.sudoers"
|
||||||
[[ $(GOWORK=off go list -m all | wc -l) -eq 1 ]]
|
[[ $(GOWORK=off go list -m all | wc -l) -eq 1 ]]
|
||||||
git diff --check
|
git diff --check
|
||||||
echo "Tend verification passed"
|
echo "Tend verification passed"
|
||||||
|
|||||||
Reference in New Issue
Block a user