Operations

Requirement A workspace MUST start WebApi, PostgreSQL, configured frontends, and its Worker where one exists, with one local command named in the consumer AGENTS.md.

Rationale The baseline command is Aspire's AppHost, and ServiceDefaults supplies the shared discovery, health, resilience, and telemetry defaults its resources rely on. A workspace that orchestrates the same set another way satisfies this rule with its own command. The obligation is that one command brings the stack up, not that one tool does.

Requirement A host MUST emit structured logs, traces, and metrics, with correlation identifiers on logs and traces and service labels on metrics.

Rationale Logs and traces carry timestamp, level, service, environment, trace identifier, and span identifier. Metrics carry service and environment labels. Correlated identifiers let one request be followed across hosts. Stable event names keep dashboards working across releases.

Requirement A host MUST read and write trace context as the W3C Trace Context traceparent and tracestate headers, using one configured propagator.

Rationale An identifier that each host formats its own way correlates nothing across the boundary between them. traceparent is the format OpenTelemetry implements and every managed backend accepts. A trace then crosses a frontend, an API, a background host, and a provider without translation.

One propagator is the part that matters. Two configured propagators produce two identifiers for one request, and the trace splits at whichever hop reads the other one.

Example The trace identifier in a Problem Details response is the trace-id field of the current traceparent, which is what litespec/backend/api#return-stable-problem-details returns.

Requirement A host MUST expose /health/live without dependency checks and /health/ready including its critical dependencies.

Rationale Readiness fails when the application cannot safely serve traffic, while liveness stays true so the orchestrator does not restart a healthy process.

Requirement A hosted schema change MUST run as a dedicated release step before traffic shifts, reviewed against both application versions.

Rationale A replica that applies schema at startup races its siblings and serves traffic against a half-applied schema.

Requirement A release MUST document backup owner, schedule, retention, encryption, location, restore command, and recovery objectives.

Rationale A backup without a tested restore is not a backup.

Requirement A project MUST rehearse a restore on representative data before its first deployment holds production data.

Rationale The first restore attempt is the wrong time to learn the backup cannot restore.

Requirement A deployment MUST use versioned artifacts, declared configuration, a schema step, readiness checks, and a passing release job.

Rationale Deploying from an uncommitted tree or a mutable branch reference makes the deployed content unknowable afterwards.

Requirement The rollback runbook MUST state the previous artifact reference, command, configuration and schema compatibility, data recovery condition, and the expand-and-contract sequence for destructive schema work.

Rationale Rollback is a required runbook, so litespec/quality/operations#exercise-every-runbook-before-release rehearses it.

Requirement A Worker MUST publish its own liveness, readiness, processing rate, failure count, retry count, oldest pending age, and shutdown behavior.

Rationale WebApi readiness otherwise hides a failed durable-delivery Worker while the queue grows unobserved.

Requirement Every recurring job MUST report liveness that pages a person when the job goes silent.

Rationale A job that stops without a signal looks exactly like a job with nothing to do.

Requirement Every network call MUST declare an explicit timeout and cancellation path, with retries only for idempotent operations named by a bounded retry policy.

Rationale An unbounded call holds a request thread until an unrelated system recovers.

Requirement A release MUST call only providers the generated sub-processor list names from the project's provider Actors.

Rationale The model builder renders the list from the provider Actor records. A provider added in code but missing from the list answers a customer question wrong.

Requirement A baseline alert MUST declare owner, threshold, evaluation window, severity, and runbook before a production release.

Rationale The alert inventory is generated from the Limit entries and the runbook each names.

Requirement Baseline coverage MUST include sustained readiness failures, unexpected HTTP errors, release-flow latency, PostgreSQL outages, and failed deployments.

Rationale These failures end a release when nobody observes them.

Requirement Every alert MUST name the runbook that answers it.

Rationale An alert without a runbook asks whoever receives it to invent the response.

Requirement An alert inventory MUST state, for each severity it uses, the destination that receives the alert and the response time expected of it.

Rationale A declared severity with no destination is a label. Every alert then arrives in the same place, and the page that wakes somebody is read beside the one that does not need to. Prometheus alerting practice states the same separation between a page and a ticket. The limits page declares the routing beside the generated inventory.

Example A two-severity inventory routes page to the on-call destination with a stated response time, and ticket to the queue reviewed each working day.

Requirement Each required runbook MUST carry a dated rehearsal in the operations log before a production release, with restore rehearsed on representative data.

Rationale A runbook is a claim until somebody runs it. The restore rehearsal is the one litespec/quality/operations#exercise-a-restore-before-the-first-production-data requires.

Requirement Production environment values MUST live on the host, never in the repository.

Rationale A secret committed once lives in history after deletion. The repository carries an example file with placeholders, and the host carries the values.