# Scheduled Jobs

## Intent

Scheduled work runs independently of HTTP requests and remains safe when replicas restart, overlap, or lose connectivity. This extension covers recurring and delayed jobs, not durable event dispatch.

## Activation

Activation scope: `local`.

Applicable specification kinds: `use-case`, `journey`.

The consumer enables `jobs` for recurring, delayed, or calendar-based work. It activates `litespec/backend/architecture#declare-the-execution-host-for-work-that-outlives-a-request`. The project declares the execution host that runs the scheduler.

## Baseline relationship

This extension replaces no baseline rule.

This extension switches on the Limit and Runbook record types.

## Standards

### Record schedule behavior

**Requirement:** A scheduled-job record MUST name schedule, time zone, delay, overlap, misfire, idempotency, retry, owners, and status, with a draft naming decider and decide-by date.

**Rationale:** The contract describes when work may run and how it recovers from normal scheduling faults. Owners are the operating owner and the record owner.

### Run jobs in Worker

**Requirement:** A scheduled job MUST run in Worker outside WebApi request handling.

**Rationale:** Worker execution permits independent scheduling and controlled replica shutdown.

### Bound job execution

**Requirement:** A scheduled job MUST observe cancellation, have bounded duration, and stop accepting new work during graceful shutdown.

**Rationale:** Bounded work lets a Worker stop without abandoning uncontrolled execution.

### Store execution leases

**Requirement:** A scheduled-job store MUST record lease owner, expiry, and fencing value in PostgreSQL.

**Rationale:** The lease identifies the replica that can complete one planned occurrence. The outbox extension cites this definition for its own claims.

### Renew only owned leases

**Requirement:** A Worker replica MUST renew only its active lease and stop when renewal fails.

**Rationale:** Losing the lease ends authority to complete the scheduled occurrence.

### Prevent duplicate completion

**Requirement:** A scheduled-job lease MUST prevent two replicas from completing the same occurrence.

**Rationale:** One occurrence needs one logical completion even when replicas overlap.

### Generate monotonic fencing values

**Requirement:** A scheduled-job store MUST generate each fencing value monotonically per occurrence and check it on every completion write.

**Rationale:** A monotonic value lets the store reject a completion from a Worker that no longer owns the lease.

### Scope occurrences and leases by tenant

**Requirement:** A scheduled-job occurrence and its lease MUST carry tenant scope when the product isolates tenants.

**Rationale:** Tenant scope keeps one tenant's scheduled work separate from another's.

### Dispatch Application behavior

**Requirement:** A scheduled job MUST dispatch an Application Command or approved Application port.

**Rationale:** Application owns the use-case boundary that the schedule triggers.

### Exclude direct storage and HTTP execution

**Requirement:** A scheduled job MUST NOT mutate a Marten session or DbContext directly, bypass Domain behavior, or call WebApi endpoints.

**Rationale:** Direct storage and HTTP calls bypass the command boundary and its authorization, transaction, and validation behavior.

### Bound job retries

**Requirement:** A scheduled job MUST use bounded attempts, cancellation-aware backoff, and a poison state for permanent failure.

**Rationale:** Bounded retries prevent one failed occurrence from consuming Worker capacity indefinitely.

### Tolerate duplicate execution

**Requirement:** A scheduled job Command and side effect MUST tolerate duplicate execution after lease or process failure.

**Rationale:** A failed process can leave uncertain completion before its lease expires.

### Treat unavailable schedule stores as outages

**Requirement:** A Worker MUST treat an unavailable scheduler store as the store outage `litespec/ext/outbox#distinguish-dependency-outage` defines, with backed-off, rate-limited logging.

**Rationale:** Store availability differs from a claimed occurrence's retry or poison outcome.

### Persist occurrence recovery points

**Requirement:** A scheduled-job store MUST persist each occurrence's last scheduled, started, and completed time.

**Rationale:** Persisted points identify missed work and completed work after restart.

### Define misfire recovery

**Requirement:** A scheduled-job specification MUST define skipped, once-replayed, or interval-replayed behavior for missed occurrences.

**Rationale:** The selected policy controls the business effect of downtime and delay.

### Provide operational recovery procedures

**Requirement:** A scheduled-job owner MUST provide Runbook records for inspection, manual replay, and disable.

**Rationale:** Operators need controlled actions for stalled, failed, or unsafe schedules.

Each procedure names the interface that performs it. A procedure that says "disable the schedule" without naming a command, an endpoint, or a configuration value remains undiscovered until the incident it was written for. Avoid an interface that writes to the schedule store directly, because it bypasses the occurrence identity `litespec/ext/jobs#derive-stable-occurrence-identity` depends on.

**Example:** A disable procedure names the configuration value or the administrative endpoint that stops the schedule. It states what happens to an occurrence already claimed.

### Derive stable occurrence identity

**Requirement:** A scheduled-job store MUST derive each occurrence identity from its schedule ID and scheduled instant, removing occurrences older than the schedule's retention Limit record.

**Rationale:** One planned time receives one stable identity across retries and replicas.

The key space grows with the schedule's frequency and its retention. A per-second schedule kept for a year holds more than 30 million rows. [Quartz states the same practice](https://quartz-scheduler.net/documentation/best-practices.html) for a persistent job store.

**Example:** A daily schedule that retains occurrences for 90 days holds 90 rows. A minute schedule with the same retention holds more than 100,000 rows. That is a different storage decision.

### Persist occurrence state

**Requirement:** A scheduled-job store MUST persist scheduled, processing, completed, and dead-letter state with safe execution details.

**Rationale:** State records include attempt, lease, start, completion, and safe error data.

### Scope command idempotency by occurrence

**Requirement:** A scheduled job MUST use its occurrence identity as the command idempotency scope.

**Rationale:** Retries of one occurrence remain tied to the same scheduled business action.

### Restrict completion by fencing value

**Requirement:** A Worker MUST mark completion only while its fencing value remains current.

**Rationale:** A stale Worker does not complete work after another replica owns the occurrence.

### Audit manual replay

**Requirement:** A manual replay MUST create a new attempt for the same occurrence identity: an audit record when audit is selected, a standalone record otherwise.

**Rationale:** Replay preserves the original scheduled instant and its history.

### Prefer UTC schedules

**Requirement:** A scheduled job MUST use a UTC schedule unless its specification names a business-local time requirement and its IANA time zone.

**Rationale:** A local schedule runs twice on one date each year and skips an hour on another. A daylight-saving transition does that to a wall clock. That behavior is correct for a job whose meaning is local, such as a nightly statement at 9 AM for a reader. It is a defect for every other job. Naming the time zone in the specification makes the transition behavior reviewable.

### Define business-local clock behavior

**Requirement:** A business-local schedule MUST record an IANA time-zone identifier and define skipped and repeated local-time behavior.

**Rationale:** Clock transitions can omit or repeat a local wall-clock time.

### Test time transitions

**Requirement:** A scheduled-job test MUST use an injected clock and representative time-zone transitions.

**Rationale:** A controllable clock exercises delayed and daylight-saving behavior deterministically.

### Exclude machine-local time

**Requirement:** Scheduled-job production code MUST NOT depend on machine-local time.

**Rationale:** Host-local configuration makes schedule behavior differ between environments.

## Conventions

### Place handlers with owned behavior

**Default:** Create one job handler under the module that owns its behavior.

**Replacement:** A consumer records a departure as an Exception record with reason and review date.

**Rationale:** The module owns the business action and its schedule meaning.

### Use execution scopes

**Default:** Resolve scoped dependencies inside an execution scope and dispose the scope after the Command completes.

**Replacement:** A consumer records a departure as an Exception record with reason and review date.

**Rationale:** One scope isolates a job occurrence's dependencies and disposal.

### Separate schedule definitions

**Default:** Keep schedule definitions separate from Command behavior.

**Replacement:** A consumer records a departure as an Exception record with reason and review date.

**Rationale:** Timing policy and business behavior change for different reasons.

## Dependencies

This extension requires no additional baseline package. Schedule state uses selected persistence and PostgreSQL leases.

## Verification

| ID | Mark | Evidence |
|:---|:---|:---|
| litespec/ext/jobs#record-schedule-behavior | guidance | Does each schedule record name every required field, its owner, and its status? |
| litespec/ext/jobs#run-jobs-in-worker | guidance | Does every scheduled job run in Worker, outside WebApi request handling? |
| litespec/ext/jobs#bound-job-execution | guidance | Does each job honor cancellation and duration bounds and stop during shutdown? |
| litespec/ext/jobs#store-execution-leases | guidance | Does the store record lease owner, expiry, and fencing value per occurrence? |
| litespec/ext/jobs#renew-only-owned-leases | guidance | Does a replica stop when its lease renewal fails? |
| litespec/ext/jobs#prevent-duplicate-completion | guidance | Does the lease prevent two replicas from completing one occurrence? |
| litespec/ext/jobs#generate-monotonic-fencing-values | guidance | Are fencing values monotonic per occurrence and checked on completion? |
| litespec/ext/jobs#scope-occurrences-and-leases-by-tenant | guidance | Do occurrences and leases carry tenant scope where tenants are isolated? |
| litespec/ext/jobs#dispatch-application-behavior | guidance | Does each job dispatch an Application Command or approved port? |
| litespec/ext/jobs#exclude-direct-storage-and-http-execution | guidance | Is job source free of direct storage mutation and WebApi calls? |
| litespec/ext/jobs#bound-job-retries | guidance | Are attempts bounded with cancellation-aware backoff and a poison state? |
| litespec/ext/jobs#tolerate-duplicate-execution | guidance | Do the job Command and side effect stay correct when a lease or process failure replays them? |
| litespec/ext/jobs#treat-unavailable-schedule-stores-as-outages | guidance | Does an unavailable store get backed-off, rate-limited logging? |
| litespec/ext/jobs#persist-occurrence-recovery-points | guidance | Does the store persist last scheduled, started, and completed times? |
| litespec/ext/jobs#define-misfire-recovery | guidance | Does the specification define skip or replay behavior for missed occurrences? |
| litespec/ext/jobs#provide-operational-recovery-procedures | guidance | Do Runbook records cover inspection, manual replay, and disable? |
| litespec/ext/jobs#derive-stable-occurrence-identity | guidance | Are occurrence identities stable per schedule and instant, with declared retention? |
| litespec/ext/jobs#persist-occurrence-state | guidance | Does the store persist scheduled, processing, completed, and dead-letter state? |
| litespec/ext/jobs#scope-command-idempotency-by-occurrence | guidance | Do retries of one occurrence share its idempotency scope? |
| litespec/ext/jobs#restrict-completion-by-fencing-value | guidance | Is completion refused once the fencing value is stale? |
| litespec/ext/jobs#audit-manual-replay | guidance | Does manual replay create a new audited attempt under the same identity? |
| litespec/ext/jobs#prefer-utc-schedules | guidance | Is each schedule UTC or a documented business-local deviation? |
| litespec/ext/jobs#define-business-local-clock-behavior | guidance | Does each local schedule name its zone and transition behavior? |
| litespec/ext/jobs#test-time-transitions | guidance | Do tests inject the clock and cover representative transitions? |
| litespec/ext/jobs#exclude-machine-local-time | guidance | Is production job code free of machine-local time reads? |
| litespec/ext/jobs#place-handlers-with-owned-behavior | guidance | Does each handler sit with its behavior's module or record an Exception? |
| litespec/ext/jobs#use-execution-scopes | guidance | Does each occurrence resolve scoped dependencies in one disposed scope? |
| litespec/ext/jobs#separate-schedule-definitions | guidance | Are schedule definitions separate from Command behavior or is an Exception recorded? |