Jobs and Queues

The job record states when the work runs: every or cron with its timezone, so tools read the rhythm instead of parsing prose. A schedule kept outside the record runs with no recorded moment attached, and misfire behavior has no stated time to measure against. The automation page states the record fields, and the schedule-invalid check reports a schedule no tool can read.

Checked bylitespec model check reports schedule-invalid (level G) for a job schedule that is neither an every duration nor a cron expression; each scheduled job record states its timezone.

Marked checkable: the tool named here checks it.

The selection states what each run selects, the batch size bounds how many items one batch takes, and runs names the command run per item. Each item runs as one command in its own unit of work, as litespec/backend/application#issue-per-record-work-from-a-job requires: a failed item replays without replaying its batch.

Reviewed forEach job run runs one command per item in its selection within its batch size, each in its own unit of work.

Marked guidance: a reviewer answers this question.

A batch that fails halfway resumes instead of restarting: the cursor holds where the run stopped, beside the saga state it drives. Without it, the next run replays finished items or skips unfinished ones, and a scenario covering the remainder resumes has no stated position to assert.

Reviewed forEach job driving a saga names its resume position in its cursor, with a scenario covering the remainder after a halfway failure.

Marked guidance: a reviewer answers this question.

Manual work is interactive platform work with no aggregate, such as a rehearsal harness, so no model bounds where it runs or who runs it. runsWhere names the deployment scope it may touch and the guard names who may run it and when. The automation page states the record fields, and the job-guard-missing check reports a manual job with no guard.

Checked bylitespec model check reports job-guard-missing (level G) for a manual job with no guard; each manual run stays inside its runsWhere.

Marked checkable: the tool named here checks it.

Scheduled work runs in a Worker, outside WebApi request handling. The Worker owns the schedule's lifetime and shuts its replicas down in a controlled order. Request handling has neither a scheduler nor a shutdown order for background work.

Reviewed forWorker sources hold every scheduled job, and WebApi request handling holds none.

Marked guidance: a reviewer answers this question.

Each job observes cancellation, so an occurrence ends its current step instead of running past the moment an operator asked it to stop.

Reviewed forEach job handler threads cancellation into its waits and stops its current step when cancellation fires.

Marked guidance: a reviewer answers this question.

Each job bounds its duration, so a Worker stops without stranding work it can no longer watch.

Reviewed forEach job handler carries a duration bound, and the occurrence store shows completions inside it.

Marked guidance: a reviewer answers this question.

Each job stops accepting new work during graceful shutdown. An unbounded job turns every deploy into a wait for work that never ends.

Reviewed forThe Worker shutdown path stops schedule polling before it drains running occurrences.

Marked guidance: a reviewer answers this question.

Each planned occurrence carries a lease with its owner, expiry, and fencing value in PostgreSQL. The lease names the one replica that may complete that occurrence. The lease field on the job record names the lock each run claims. The outbox extension cites this definition for its own claims.

Reviewed forThe lease rows in PostgreSQL carry owner, expiry, and fencing value for each occurrence.

Marked guidance: a reviewer answers this question.

A replica renews only the lease it holds and only while it stays active. A failed renewal means another replica owns the occurrence now. The loser stops instead of completing work it no longer owns.

Reviewed forReplica logs show no renewal after a failed renewal, and the loser stops its occurrence.

Marked guidance: a reviewer answers this question.

Overlapping replicas must not complete the same occurrence twice. The lease admits one completion per occurrence. Two completions would bill, notify, or reconcile the same planned work twice.

Reviewed forThe lease admits one completion per occurrence across overlapping replicas.

Marked guidance: a reviewer answers this question.

Each fencing value rises monotonically per occurrence, so a later claim always presents a higher value than an earlier one. A Worker that lost its lease presents a stale value, and litespec/ext/jobs#restrict-completion-by-fencing-value refuses completion on a stale value.

Reviewed forFencing values in the store rise per occurrence, and a stale value refuses completion.

Marked guidance: a reviewer answers this question.

Products that isolate tenants scope each occurrence and its lease to one tenant. One tenant's schedules then never trigger, block, or expose another tenant's work. Unscoped leases would let a replica mix tenants inside one occurrence.

Reviewed forOccurrence and lease rows carry tenant scope in products that isolate tenants.

Marked guidance: a reviewer answers this question.

The job record names the use cases it runs, and the use case boundary lives in Application. Each job therefore dispatches an Application Command or a declared Application port. The schedule stays a trigger while the behavior stays owned.

Reviewed forEach job handler calls an Application Command or a declared Application port.

Marked guidance: a reviewer answers this question.

A job that mutates a Marten session or DbContext directly, bypasses Domain behavior, or calls WebApi endpoints skips the command boundary. That boundary carries authorization, transaction, and validation the job would then lack. Jobs enter through Commands like every other caller.

Reviewed forJob sources mutate no Marten session or DbContext and call no WebApi endpoint.

Marked guidance: a reviewer answers this question.

Bounded attempts with cancellation-aware backoff cap what one failed occurrence consumes. A poison state parks the permanently failing occurrence instead of retrying it forever. Without the bound, one poison occurrence eats the Worker's whole capacity.

Reviewed forEach schedule declares bounded attempts with cancellation-aware backoff and a poison state.

Marked guidance: a reviewer answers this question.

A lease or process failure can replay a job whose completion stayed uncertain. The job Command and its side effects tolerate that duplicate execution. Work that assumes exactly-once breaks on the first uncertain failure.

Reviewed forJob Commands and side effects show one accepted effect after a replayed occurrence.

Marked guidance: a reviewer answers this question.

An unavailable scheduler store is an outage of the dependency, not a failed occurrence. The Worker treats it as the store outage litespec/ext/outbox#distinguish-dependency-outage defines, with backed-off, rate-limited logging. Retry and poison outcomes belong to claimed occurrences, not to a store nobody can reach.

Reviewed forWorker logs during a store outage show backed-off, rate-limited entries and no poisoned occurrences.

Marked guidance: a reviewer answers this question.

Last scheduled, started, and completed times persist per occurrence. After a restart, those points separate missed work from completed work. Memory alone cannot make that split, because the restart wipes it.

Reviewed forEach occurrence row carries last scheduled, started, and completed times.

Marked guidance: a reviewer answers this question.

Downtime and delay leave occurrences past their time. The specification states whether each missed occurrence is skipped, replayed once, or replayed per interval. The choice controls the business effect: a skipped reminder stays silent while a replayed one arrives late.

Reviewed forEach schedule states skipped, once-replayed, or interval-replayed behavior for missed occurrences.

Marked guidance: a reviewer answers this question.

Stalled, failed, or unsafe schedules need controlled operator actions: inspection, manual replay, and disable. Runbook records carry those procedures. An operator without them improvises against the store during the incident.

Each procedure names the interface that performs it. A procedure that says "disable the schedule" without naming a command, an endpoint, or a configuration value remains undiscovered until the incident it was written for. Avoid an interface that writes to the schedule store directly, because it bypasses the occurrence identity litespec/ext/jobs#derive-stable-occurrence-identity depends on.

A disable procedure names the configuration value or the administrative endpoint that stops the schedule. It states what happens to an occurrence already claimed.

Reviewed forRunbook records cover inspection, manual replay, and disable for each schedule.

Marked guidance: a reviewer answers this question.

Schedule ID plus scheduled instant gives each planned time one stable identity across retries and replicas. Retries of one occurrence then share their idempotency scope, and two replicas recognize the same occurrence.

Reviewed forOccurrence identities in the store combine schedule ID with scheduled instant.

Marked guidance: a reviewer answers this question.

Occurrences older than the retention setting record leave the store so the key space stays bounded.

The key space grows with the schedule's frequency and its retention. A per-second schedule kept for a year holds more than 30 million rows. Quartz states the same practice for a persistent job store.

A daily schedule that retains occurrences for 90 days holds 90 rows. A minute schedule with the same retention holds more than 100,000 rows. That is a different storage decision.

Reviewed forThe store holds no occurrence older than its schedule's retention setting record.

Marked guidance: a reviewer answers this question.

Scheduled, processing, completed, and dead-letter states persist with attempt, lease, start, completion, and safe error data. The persisted state tells operators and recovery code where each occurrence stands. A store without states cannot separate work in flight from work never started.

Reviewed forEach occurrence row carries scheduled, processing, completed, or dead-letter state with safe execution details.

Marked guidance: a reviewer answers this question.

The occurrence identity scopes the command idempotency key. Every retry of one occurrence then replays the same scheduled business action. A key scoped any wider would merge separate occurrences into one.

Reviewed forRetries of one occurrence share the idempotency scope of its occurrence identity.

Marked guidance: a reviewer answers this question.

The Worker marks completion only while its fencing value stays current. A stale value means another replica owns the occurrence now. Completing past that point would write one replica's result over another's ownership.

Reviewed forCompletion writes fail on a stale fencing value in the store.

Marked guidance: a reviewer answers this question.

A manual replay creates a new attempt under the same occurrence identity: an audit record where audit is selected, a standalone record otherwise. The replay keeps the original scheduled instant and its history. A replay recorded as a new occurrence would rewrite when the work was due.

Reviewed forEach manual replay adds an audited attempt under its original occurrence identity.

Marked guidance: a reviewer answers this question.

A local schedule runs twice on one date each year and skips an hour on another. A daylight-saving transition does that to a wall clock. That behavior is correct for a job whose meaning is local, such as a nightly statement at 9 AM for a reader. It is a defect for every other job. Naming the time zone in the specification states the transition behavior for a later reader.

Reviewed forEach schedule states UTC or names a business-local time requirement with its IANA time zone.

Marked guidance: a reviewer answers this question.

Daylight-saving transitions omit one local hour and repeat another. A business-local schedule records its IANA time-zone identifier and defines what happens to occurrences in the skipped and repeated hours. Without that definition, each transition silently drops or doubles the local work.

Reviewed forEach business-local schedule records its IANA zone with skipped and repeated local-time behavior.

Marked guidance: a reviewer answers this question.

An injected clock lets tests move time to any transition on demand. Representative time-zone transitions then run deterministically instead of once a year in production. Tests on the real clock can only wait for the transition to arrive.

Reviewed forJob tests inject the clock and cover representative time-zone transitions.

Marked guidance: a reviewer answers this question.

Machine-local time follows each host's configuration. Production code that reads it schedules differently on a developer machine, in staging, and in production. Schedules read their time from the injected clock and the declared zone instead.

Reviewed forProduction job sources read no machine-local clock.

Marked guidance: a reviewer answers this question.

Two people clearing the same item double-pay, double-approve, or double-refuse. The claim mode says whether items need no claim, accept an optional hold, or require one before action, so the queue never leaves two hands on one item.

The payout queue requires a claim before release, while the review queue leaves claiming optional.

Reviewed forThe queue use cases state disabled, optional, or required on their claim field.

Marked guidance: a reviewer answers this question.

A queue without an order is cleared by habit: newest first, loudest first, easiest first. A score citing a calculation rule pins the order to declared inputs, so the riskiest or most valuable item surfaces first by rule instead of by accident.

The dispute queue orders by rule/dispute.urgency-score, which weighs amount, age, and evidence gaps.

Reviewed forThe queue listings sort by a score citing the calculation rule behind it.

Marked guidance: a reviewer answers this question.

A hidden score is an oracle: people work around an order they cannot see or trust. Showing the score with its inputs lets each person check the order and challenge the inputs instead of the queue.

Each dispute row shows urgency 82: EUR 1,900, 6 days old, evidence missing.

Reviewed forThe queue rows display each score with the inputs that set it.

Marked guidance: a reviewer answers this question.

Claims are held by people who go home: a claim with no expiry parks the item forever. The lease returns stale claims to the queue, so an abandoned item waits for the next person instead of for nobody.

A payout claimed Friday and untouched returns to the queue when its one-hour lease expires.

Reviewed forThe claim tests show untouched items returning after their lease expires.

Marked guidance: a reviewer answers this question.

Bulk money moves on many items at once, and one wrong filter multiplies one mistake into hundreds. The preview lists every affected item with its amount, so the person confirms the set they meant instead of the set they typed.

The bulk release previews forty-two payouts totaling EUR 18,400 before asking for confirmation.

Reviewed forThe bulk money commands render every affected item with its amount before acting.

Marked guidance: a reviewer answers this question.

A preview confirms the set, but the session itself may be hijacked, shared, or left open. An out-of-band code arrives on a channel the session does not control, so the confirmation proves the person, not just the browser.

The bulk release asks for the six-digit code sent to the finance phone before moving any money.

Reviewed forThe bulk money commands ask for the out-of-band code before moving money.

Marked guidance: a reviewer answers this question.

Order is the pipeline's meaning: enriching before dropping wastes work, and dropping before enriching loses context. Listing the steps in run order with each kind marked lets a reader replay the pipeline in their head.

The catalog ingest lists drop-duplicates, then enrich-prices, then drop-below-threshold.

Reviewed forThe ingest job records list steps in run order with each marked drop or enrich.

Marked guidance: a reviewer answers this question.

A dropped record with no reason is indistinguishable from a lost one. A closed set forces each drop to say why in words the team chose, so operators count drops by reason instead of guessing.

The catalog ingest names duplicate, below-threshold, and poison as its only drop reasons.

Reviewed forThe ingest job records name the closed drop reasons with no drop outside them.

Marked guidance: a reviewer answers this question.

The set alone promises nothing until each drop cites it. One reason on each dropped record turns the set into evidence: every drop answers why, in the team's own words.

A feed row dropped for matching an existing SKU carries duplicate.

Reviewed forThe dropped records each carry one reason from the job's closed set.

Marked guidance: a reviewer answers this question.

Enrichment pulls outside data into product records: prices, categories, ratings. An undeclared source smuggles values no record explains, so each enrich step reads only the sources its job names.

The enrich-prices step reads the vendor price list its job declares, and nothing else.

Reviewed forThe enrich steps read only the sources named on their job records.

Marked guidance: a reviewer answers this question.

Feeds burst: a vendor resends a week of rows, a file lands twice. The buffer holds the surge, and the policy states how much it holds and what happens past that: block the feed, drop the oldest, or page an operator. Without the policy, the first surge picks the behavior.

The catalog ingest buffers ten thousand rows and blocks the feed past that until the buffer drains.

Reviewed forThe ingest job records state buffer capacity with the overflow behavior past it.

Marked guidance: a reviewer answers this question.

Drop counts are the pipeline's pulse: a spike in poison means a broken feed, a spike in duplicate means a double send. Per-reason counts turn each drop into a signal the team watches.

The catalog dashboard shows twelve poison drops in the last hour beside zero duplicate drops.

Reviewed forThe pipeline dashboards show dropped-record counts split by reason.

Marked guidance: a reviewer answers this question.

Ingested data lags: the buffer, the batch window, and the enrich calls all take time. The query states what its readers observe, a bounded object with its window or an advisory object with its authority, so a stale price reads as declared behavior instead of a defect.

The catalog search query states {"mode": "bounded", "within": "PT15M"}, matching the ingest job's quarter-hour window.

Reviewed forThe queries over ingested data state the freshness their readers observe.

Marked guidance: a reviewer answers this question.

A reader meeting a saga asks first what moves it: reactions, jobs, or an engine. The carrier marker answers that question on the aggregate itself, so nobody hunts the engine console for a saga reactions and jobs drive.

The hold aggregate sets carriedBy: workflow while the loan aggregate stays driven by reactions and jobs.

Reviewed forThe engine-carried aggregates set carriedBy workflow on their aggregate records.

Marked guidance: a reviewer answers this question.

An engine step with no use case is behavior no record describes, and a step mapped to two use cases is two behaviors wearing one name. One step and one use case keep the engine definition and the records in lockstep: the map reads both ways.

The engine step cancel-hold maps to use-case/lending.cancel-hold and to nothing else.

Reviewed forThe engine definition lists one use case beside each step with no step sharing or missing.

Marked guidance: a reviewer answers this question.

Engines fail mid-saga: a step times out, a worker crashes, a saga aborts. The compensation is the step's undo as {"mode": "use-case", "useCase": "use-case/..."}, and a step with no undo says so as {"mode": "none", "reason": "..."}, so recovery follows the record instead of improvising.

The step cancel-hold names {"mode": "use-case", "useCase": "use-case/lending.place-hold"} as its compensation, while borrow-copy names {"mode": "none", "reason": "the copy has left the shelf"}.

Reviewed forWhich mapped steps lack a compensation object naming its use case or its reason? Look at the map beside the process.

Marked guidance: a reviewer answers this question.

The map is the whole contract of this extension: every step answers to a use case. A step outside the map is unrecorded behavior by definition, so it fails the same check as code with no record.

A new engine step added in the console with no use case behind it fails behavior-unrecorded at the next check.

Checked bylitespec model check reports behavior-unrecorded for an engine step that maps to no use case.

Marked checkable: the tool named here checks it.

Retry counts, backoff curves, and step timeouts tune operations, not behavior: they change with load and incidents, not with product decisions. Records state what each step does and undoes, while code holds the knobs operators turn.

The engine's five-attempt retry with exponential backoff lives in the worker configuration, while the records name only the step and its compensation.

Reviewed forRetry counts, backoff curves, and step timeouts live in worker code with none in the records.

Marked guidance: a reviewer answers this question.

Reporting infrastructure costs a Worker, storage, and operating attention, so the team writes down the report, a representative data volume, the plan or limitation, the latency target, the maximum result, and the operator before building any of it. The use case page carries that record, and it shows why a normal request projection cannot meet the boundary.

Reviewed forThe use case page holds the report, volume, plan, target, maximum, and operator.

Marked guidance: a reviewer answers this question.

A reporting system built for analytics the product may never need charges its full operating cost from day one while serving no accepted query. Each report or export earns its infrastructure with a current cost boundary, not a guess about future analysis.

Reviewed forEach reporting system traces to an accepted current query or export.

Marked guidance: a reviewer answers this question.

A report concatenates actor, tenant, filters, ranges, sorting, and pagination into SQL that reads across the database. Binding every one of those values as a parameter keeps tenant and actor isolation intact and closes the injection hole string concatenation opens.

Reviewed forEach report query binds actor, tenant, filter, range, sort, and pagination values as parameters in code.

Marked guidance: a reviewer answers this question.

Raw SQL names tables, dialects, and connection handling that belong to one provider. Keeping it in Infrastructure lets Application and Domain state what the report means without depending on how one database phrases it.

Reviewed forRaw report SQL and connection handling sit in Infrastructure code.

Marked guidance: a reviewer answers this question.

A query that flies over a hundred local rows can scan millions in production. The team reviews query plans and indexes against representative data, so the true cost shows before the report ships rather than after it slows the database.

Reviewed forThe review record shows query plans and indexes checked against representative data.

Marked guidance: a reviewer answers this question.

A report that reads ten thousand rows at once crosses the same actor, role, ownership, tenant, and sensitive-field boundaries as ten thousand single reads. Enforcing the item-read rules on the bulk path stops the report from exposing rows its reader may not open one by one.

Reviewed forEach report and export enforces the item-read rules in the authorization tests.

Marked guidance: a reviewer answers this question.

An export that outlasts its request budget holds an HTTP connection open while the reader waits and the server counts threads. Running it in a Worker returns the request immediately, and the budget in its setting record tells the team exactly when an export outgrows the request path.

Reviewed forEach export past its request budget runs through a Worker, with the budget in its setting record.

Marked guidance: a reviewer answers this question.

An export file concentrates sensitive rows into one downloadable object that outlives the request that built it. Approved object storage with a named contact, an expiry, and a short-lived authorized download gives that object controlled access and a defined end instead of a permanent public link.

Reviewed forExport output sits in approved object storage with a named contact, an expiry, and a short-lived authorized download.

Marked guidance: a reviewer answers this question.

Export addresses are predictable, and a shared folder lets one tenant guess another tenant's file name. A tenant-scoped storage path puts each tenant's exports under its own prefix, so a guessed address stays inside the guesser's own data.

Reviewed forEach export stores its output under a tenant-scoped path in object storage.

Marked guidance: a reviewer answers this question.

Date range, rows, execution time, concurrent jobs, output size, and storage lifetime each cap one dimension of report cost. Setting records hold all six, so a runaway report hits a documented bound instead of an exhausted database, a full disk, or a surprise invoice.

Reviewed forEach report bound lives as a setting record beside the report.

Marked guidance: a reviewer answers this question.

An export reads many rows at once, so it states how the read is bounded: {"mode": "pages", ...} bounds each run, and {"mode": "none", "reason": "..."} states why the export reads all rows at once. Without the declaration, a reviewer cannot tell a bounded export from one that grows with the table.

Reviewed forWhich export queries lack paging with mode pages or mode none with its reason? Look at the export query beside the report.

Marked guidance: a reviewer answers this question.

An export file concentrates sensitive rows into one downloadable object, so the use case states why the bulk read is audited and where the audit lands. Without the stated audit, the concentrated read leaves no trace a reviewer can follow.

Reviewed forWhich export queries lack a stated audit reason and target? Look at the export query beside the report.

Marked guidance: a reviewer answers this question.

A reader who closes the tab leaves the report running, and an export nobody downloads still occupies storage. Cancellation stops the abandoned work, and expiry removes the abandoned file, so neither consumes resources indefinitely.

Reviewed forReport work cancels on request and expired output clears in the cancellation tests.

Marked guidance: a reviewer answers this question.

A value starting with =, +, -, or @ becomes a live formula when the export opens in a spreadsheet, running in the opener's application with the opener's data. Escaping those values keeps data as data, except in a field the team intentionally fills with a formula.

Reviewed forExported values starting a formula are escaped, except in formula fields the team fills intentionally.

Marked guidance: a reviewer answers this question.

An export without a stated encoding arrives garbled on the first machine that guesses differently, and columns that shift per release break every parser built on them. Explicit encoding with a stable column contract lets consumers parse the file without guessing either.

Reviewed forEach export states its character encoding and stable columns in its format record.

Marked guidance: a reviewer answers this question.