refactor(deployment): use durable sequential app deploys (#284)

* refactor(deployment): use sequential service replacement

Replace the rolling-versus-interrupted split for app services with one
sequential replacement applied to every service:

  preflight -> withdraw traffic (confirmed) -> retire the superseded
  container (confirmed gone) -> create and start the replacement ->
  readiness probe -> publish ACTIVE -> rebuild traffic

A deploy may briefly interrupt a service, and two Gordon-managed
generations of one service never run at the same time.

* refactor(deployment): drop httpEligible, deployHTTP, deployInterrupted,
  cutoverService, retireContainerID, drainWithDeadline and the
  retire-after-publish path
* refactor(deployment): restart always restarts the same pinned container
  in place (withdraw, restart, readiness, republish) and no longer creates
  a second container
* refactor(deployment): resolve the implicit HTTP readiness probe from the
  service spec so readiness no longer depends on how a service is
  replaced, keeping the check for stateless public HTTP services
* refactor(deployment): remove DeployResult.Interrupted and
  ServiceResult.Retire/RetireGrace
* test(deployment): replace the rolling-behaviour tests with sequential
  replacement, in-place restart, readiness-ordering, withdrawal-failure,
  cancellation, recovery-inhibition and multi-service failure tests
* docs: describe sequential replacement, the expected short interruption,
  preflight protection and manifest-based rollback

* fix(deployment): recover interrupted replacements safely

Close the crash window left by sequential replacement and the recovery
dead ends the review of it exposed.

* fix(deployment): journal the replacement's container ID before it is
  started, and keep it in the journal when a replacement fails after
  creating its container, so a leftover stays traceable
* fix(deployment): reconcile an unfinished operation at boot and before
  every mutation of an app: remove a never-published candidate (releasing
  its loopback claims) before creating or rebuilding a generation, keep
  the published generation, retry the leftovers of an operation that
  already failed, and finalize the operation (failure, or success when
  every step had already completed) so a keyed retry never reads a
  permanent in-flight claim
* fix(deployment): refuse a new journal while a different operation of the
  same app is unfinished, so a claim can never mask an interrupted
  predecessor
* fix(deployment): a restart that rebuilds a missing generation clears the
  recovery inhibition it wrote for the container proven gone, which
  otherwise refused every later start, recovery pass, and restart
* refactor(deployment): recover the store before reconciling in deploy,
  move the preflight store recovery to its callers, and extract
  runtimeImageRef, convergeCandidate, and interruptedOutcome
* refactor(config): remove the dead [deploy] configuration surface
  (pull_policy, readiness_*, health_timeout, stabilization_delay, probe
  timeouts, drain_*) from the viper defaults, the example configuration,
  and the docs: no code ever read these keys
* test(deployment): cover the candidate journal write before start, boot
  and mutation-time reconciliation, terminal-operation leftovers, an
  interruption after the last step, the claim guard, restart rebuilding a
  missing container, and the inhibition cleanup
* docs: document crash recovery, the 15 s republish cadence, and restart
  rebuilding from the pinned digest

* fix(deployment): fail closed on orphan replacements and repair recovery

Close the recovery findings left by the sequential-replacement review.

* fix(deployment): fail closed when a replacement candidate cannot be removed.
  Reconciliation returns an error, keeps the journal (and its latest position)
  instead of finalizing it, and no later mutation proceeds, so a newer
  operation can never mask an orphan that may still run.
* fix(deployment): clear the superseded container's stale replacement-pending
  inhibition once its never-published candidate is confirmed gone, so
  boot/start/restart can rebuild the generation ACTIVE records. The inhibition
  is kept while candidate cleanup fails, preserving the single writer.
* fix(deployment): set Service on start/restart placeholder journal steps so a
  recovered step is attributable and its inhibition can be cleared.
* fix(deployment): repair the Preflight(key)->Deploy(same key) contract:
  Preflight reconciles before resolving and never claims the caller's request
  key, so a later Deploy of the same key executes instead of replaying the
  preflight journal as a conflict.
* refactor(deployment): drop the unused bool result of verifyRunningService.
* test(deployment): cover fail-closed candidate removal, stale inhibition
  removal with a volume-owning rebuild, the repaired preflight/deploy key
  contract, and preflight reconciliation of an unfinished predecessor.
* docs: describe fail-closed reconciliation and inhibition removal.

* refactor(deployment): split deploy engine into start and execute phases

Factor the synchronous deploy engine into a durable two-phase internal
API. StartDeploy runs under the app lock: it reconciles any interrupted
predecessor, resolves the revision and runs the convergence checks,
atomically claims the request key, and persists a non-terminal journal.
It performs no image pull, pinning, or workload mutation, and reports
whether the caller newly owns the operation or is an idempotent replay.

ExecuteDeploy reacquires the app lock, loads and verifies the claimed
journal by app/op identity, runs image pull/pinning and the existing
sequential replacement, and records the terminal outcome. A terminal or
mismatched claim is never executed: it replays instead. Cancellation or
shutdown mid-execution leaves the non-terminal journal for reconciliation.

Service.Deploy stays the synchronous composition of both phases, so all
existing callers and behavior are unchanged. Existing preflight is
re-expressed as claimDeploymentLocked plus pinPreflightLocked.

* feat(apps): orchestrate two-phase deploy in the background

AppServiceImpl.Deploy now claims and persists the operation through
StartDeploy and, for the key owner only, schedules ExecuteDeploy on a
daemon-owned context so the request returns the running journal promptly
and request cancellation never aborts the replacement. Duplicate keys
replay their stored journal and never spawn a second execution.

A daemon context plus WaitGroup on AppServiceImpl gives the lifecycle
its graceful shutdown hook: Shutdown cancels in-flight executions,
which leave the persisted non-terminal journal for reconciliation, and
waits for them to unwind.

* feat(app): wire daemon lifecycle context into app administration

Build AppServiceImpl through newAppDaemonService with the
daemon/supervisor lifecycle context, so background deploy executions are
owned by the daemon and never by a request context.

Graceful shutdown registers AppServiceImpl.Shutdown on the bounded 30s
shutdown context as Phase 0.5, cancelling and joining in-flight deploy
executions before traffic, runtime, or the app state store is torn down.
A failed unwind is logged and never aborts teardown.

* feat(apps): answer 202 for running deploys and quiesce kernel app admin

The admin deploy handler now answers 202 Accepted with the durable running journal when it newly owns the operation, so the request returns promptly and the client observes completion via GET operations/by-key. A terminal replay stays 200, and conflicts, preflight failures, and failed replays keep their existing 409 mapping. AppDeployResponse carries a stable status field (running until terminal, then the persisted outcome); the by-key lookup keeps answering 200 and remains the recovery source. Permission checks and Idempotency-Key validation are unchanged, and the remote client's generic 2xx path already accepts 202.

Kernel.Close now cancels and joins the daemon-owned AppServiceImpl on a bounded context before any remaining kernel resource is torn down, so a background deploy execution cannot outlive the state and runtime it uses. A nil implementation is never stored in the lifecycle interface.

* feat(apps): poll 202 deploy operations and add operations watch

`gordon apps deploy` now follows a backend 202 through the existing
operations/by-key endpoint until the operation is terminal, printing
concise progress only when operation or step state changes. JSON mode
keeps stdout to a single final document (progress goes to stderr) and
terminal partial/failed outcomes exit nonzero.

Add `gordon apps operations watch APP --key KEY` to resume polling an
existing operation without reissuing the mutation. Ctrl-C stops only
local polling, never the daemon-side operation, and prints the resumable
command in human mode. 404 fails fast; transient polling errors are
bounded; local and remote planes share the same client path.

Docs: apps deploy/operations sections and CLI index.

* fix(deployment): keep live deploy claims out of foreground reconciliation

Track the operations this process owns between the StartDeploy claim and the
ExecuteDeploy exit, so a foreground mutation landing in the hand-off window
cannot reconcile and terminalize a live claim. ExecuteDeploy now always
reloads the operation from the store by app/op identity and refuses a
terminal or mismatched record, never executing a stale in-process journal.

* fix(apps): settle deploy claims raced by shutdown before scheduling

Gate AppServiceImpl.Deploy on the daemon lifecycle before StartDeploy, so a
request arriving after shutdown began is refused with a wrapped
domain.ErrAppStateConflict and never claims a journal or answers 202. Enroll
the Deploy hand-off with the lifecycle so Shutdown waits for an in-flight claim
to be either scheduled or settled, closing the window where shutdown starts
after StartDeploy but before scheduleExecution.

When shutdown wins that race, settle the owned claim instead of dropping it:
add deployment.Service.AbandonDeploy, which releases the in-process live marker
and converges the still non-terminal journal as interrupted, so no accepted
durable claim stays marked live forever and a later claim recovers cleanly.

Add deterministic tests for stopping before a deploy and for shutdown racing
between the claim and scheduling, plus an engine test proving the settled claim
is terminal and never blocks the next owner.

* fix(apps): reuse deploy watch for accepted apply --deploy

`apps apply --deploy` now follows a 202 accepted deploy through the same
by-key watch as `apps deploy`: progress only when the operation or step
state changes, one terminal journal, and a nonzero exit for a terminal
partial/failed outcome without hiding the successful apply.

JSON mode emits exactly one final combined {apply,deploy} document and
sends progress, transient warnings, and Ctrl-C resume guidance to stderr;
no initial running document is written and the deploy is never re-POSTed.
The command now threads cmd.ErrOrStderr so stderr routing is exercised.

Add focused tests for 202 to success, partial/failure with preserved
apply, unchanged-state deduplication, JSON purity, single mutation, and
Ctrl-C resume in both human and JSON modes.

* fix(apps): return journal shape for running deploy replays

A same-key deploy replay whose operation is still in flight now always answers 409 with the stored operation journal, before or after any service step has started. Only an absent or terminal journal may map a preflight error envelope, so a non-terminal replay never degrades to the mapped error response.

* refactor(deployment): remove dead exported Service.Preflight

Remove the production-dead deployment.Service.Preflight entry point and
its now-unreachable preflightLocked wrapper. No production caller and no
boundary interface exposed it; only direct tests did. The private
preflight gates (claimDeploymentLocked/pinPreflightLocked) used by
StartDeploy/ExecuteDeploy are preserved.

Delete the tests and helpers that existed only for the removed entry
point.

* test(deployment): restore preflight gate coverage through deploy phases

Retarget the live preflight gates that were only reachable through the removed Service.Preflight onto StartDeploy/ExecuteDeploy and the synchronous Deploy composition: missing required secret (ErrAppSecretMissing), env/secret collision (ErrInvalidAppSpec), unmanaged image volume (ErrAppUnmanagedImageVolume), R2 unowned existing volume refusal (ErrAppStateConflict), the targeted-deploy diff policy, and the shared GC barrier held across resource selection.

Every failing gate proves no workload mutation before the failure, and the execute-phase gates prove a terminal failed journal. No production code changes.

* fix(deployment): address HTTP readiness at declared container port

qBittorrent rejects the readiness probe with 401 "Invalid Host header, port mismatch" because the probe dials the random loopback published port and Go derives the Host header from it. Keep dialing the published bind but send 127.0.0.1:<declared container readiness port> as the Host authority, matching what the workload listens on.

Thread the Host authority through the injectable httpGet dependency so tests can assert both the dial URL and the Host header, and cover the mismatch with a server that rejects published-port authorities.

* fix(deployment): harden sequential deploy lifecycle

Address the review findings on the sequential service replacement work:

- keep the revision resolved at claim time authoritative for the whole
  execution instead of re-resolving desired state in the second phase
- record successful candidate convergence by clearing the step After and
  keeping the container id in Detail, so a converged step is not retried
  while its audit trail survives
- normalize the persisted service-step plan when a claim is resumed and
  converge recorded candidates before the plan is replaced
- fail closed when app administration does not quiesce: skip state and
  runtime teardown and propagate the error to the process entry point
- carry app and op fields through zerowrap contexts for the deploy
  scheduling logs instead of the service logger
- move the operations watch command into apps_operations_watch.go
- exercise the new admin deploy handler cases over a real httptest server

commit

9ee3e24246214643171d754a5f228c663b145cbf

author

bnema · 39d ago · 2026-09-13 18:16 UTC

committer

GitHub · 39d ago · 2026-09-13 18:16 UTC

parent

a6eb480

signature

GPG UnverifiedB5690EEEBB952194 The key is unknown