Documentation menu

Getting started

New in v2.0

Feature deep dives

System atlas

Reference

Learn

Task lifecycle

Durable task lifecycle

How long-running work survives a restart: the six states, the allowed transitions, idempotent submission, retry backoff, refusal codes and the 64 KiB bounds.

Verified against the checkout built for v2.2.4 · updated 20 Oct 2026

On this page

Why a task queue
States and transitions
Task kinds
Idempotent submission
Attempts and backoff
Bounds and refusals
Walkthrough

A session backup, a dead-letter replay, a whole-installation export: each of these outlives the HTTP request that starts it. Rather than holding a connection open, the API writes a row and returns it. The row is the contract — poll it, cancel it, retry it — and it is owned by the same tenant rule as everything else, which is why a foreign task id answers exactly like one that does not exist.

Why a task queue instead of a long request

·Restarts do not lose work. The row is in the database, so a container replaced mid-backup comes back to a claimable PENDING job rather than an orphaned promise.
·Progress is observable. GET /api/tasks shows the state, the attempt count and the last error, so "is it still running?" is a query, not a guess.
·Failures are recoverable. A transient Telegram or network failure reaches RETRY_WAIT with backoff; a spent budget reaches FAILED, where an operator can retry it deliberately.
·Retries are safe. The idempotency rule means a client that retries a submission gets the original job back, not a second copy of the work.

States and transitions

PENDING
Accepted and stored. Any worker may claim it; nothing has run yet.
cancellable
RUNNING
A worker holds the claim and is executing. The attempt counter has already been incremented.
running_not_cancellable
RETRY_WAIT
The attempt failed in a way that is worth retrying. The backoff delay is counted down, and the job stays cancellable while it waits.
cancellable
SUCCEEDED
The work finished and its result is stored with the row. Terminal.
not_cancellable
FAILED
The attempt budget is spent, or the failure was not worth retrying. Terminal — unless the owner explicitly retries it.
retryable
CANCELLED
The owner stopped it before it ran, or while it waited for a retry. Terminal.
not_cancellable

The complete transition table — anything not listed here is refused
From	To	Trigger
PENDING	RUNNING, CANCELLED	A worker claims it, or the owner stops it before the claim.
RUNNING	SUCCEEDED, FAILED, CANCELLED, RETRY_WAIT	The one state with four exits: done, permanent failure, a cancelled-and-cooperative run, or another attempt later.
RETRY_WAIT	PENDING, RUNNING, CANCELLED	The delay expires and the job is queued again, or the owner stops it while it waits.
FAILED	PENDING	Only an explicit retry re-queues it, and only from FAILED.
SUCCEEDED	—	Terminal. Nothing transitions out.
CANCELLED	—	Terminal. Nothing transitions out.

PENDING
To
RUNNING, CANCELLED
Trigger
A worker claims it, or the owner stops it before the claim.
RUNNING
To
SUCCEEDED, FAILED, CANCELLED, RETRY_WAIT
Trigger
The one state with four exits: done, permanent failure, a cancelled-and-cooperative run, or another attempt later.
RETRY_WAIT
To
PENDING, RUNNING, CANCELLED
Trigger
The delay expires and the job is queued again, or the owner stops it while it waits.
FAILED
To
PENDING
Trigger
Only an explicit retry re-queues it, and only from FAILED.
SUCCEEDED
To
—
Trigger
Terminal. Nothing transitions out.
CANCELLED
To
—
Trigger
Terminal. Nothing transitions out.

Task kinds

The kinds a caller may submit
Kind	Started by	What it produces
`session.backup`	`POST /api/sessions/{id}/backup`	An encrypted archive of the session file, written only after the export succeeds.
`dlq.replay`	Dead-letter replay from the dashboard or `POST /api/tasks`	The refused deliveries re-attempted against their original targets.
`dlq.purge`	`DELETE /api/dlq`	The caller's dead-letter entries removed, ownership applied before the delete.
`backup.export`	`POST /api/backup/export`	A configuration bundle that can be imported on another host.

session.backup
Started by
POST /api/sessions/{id}/backup
What it produces
An encrypted archive of the session file, written only after the export succeeds.
dlq.replay
Started by
Dead-letter replay from the dashboard or POST /api/tasks
What it produces
The refused deliveries re-attempted against their original targets.
dlq.purge
Started by
DELETE /api/dlq
What it produces
The caller's dead-letter entries removed, ownership applied before the delete.
backup.export
Started by
POST /api/backup/export
What it produces
A configuration bundle that can be imported on another host.

Idempotent submission

The identity of a job is the trio of owner, kind and idempotency_key. The owner comes from the verified token — never from the body — so two tenants may use the same key without colliding, and a client that retried a submission after a timeout gets its original job back instead of duplicating the work.

First submission — 201 Created

{
  "task": {
    "id": "6f1c…",
    "kind": "dlq.replay",
    "status": "PENDING",
    "idempotency_key": "replay-2026-10-19",
    "attempt": 0,
    "max_attempts": 3
  },
  "already_existed": false,
  "id": "6f1c…",
  "status": "PENDING"
}

The same submission again — 200 OK, same job

{
  "task": { "id": "6f1c…", "status": "RUNNING", "attempt": 1 },
  "already_existed": true,
  "id": "6f1c…",
  "status": "RUNNING"
}

Attempts and backoff

Retry behaviour, in the order you will meet it
Rule	Value	Why
Default attempt budget	`3` attempts	Enough to ride out a transient failure, short enough that a permanent one is reported the same day.
Maximum attempt budget	`8`	A requested budget is clamped server-side rather than rejected, so an exaggerated value cannot create an unbounded retry loop.
Backoff	`5s → 20s → 80s → …`, factor 4, capped at `5 min`	Growth fast enough to stop hammering a service that is down, capped so a long queue still drains.
Attempt accounting	Incremented when the attempt starts	A worker that dies mid-attempt does not get a free retry; the attempt it was running is visible in the row.
Explicit retry	FAILED → PENDING with a fresh budget	An operator who fixed the cause re-queues the job; anything not in FAILED answers `not_retryable`.
Terminal retention	30 days by default, configurable	Finished rows are purged by a scheduled job; nothing that is still PENDING, RUNNING or RETRY_WAIT is ever eligible.

Default attempt budget
Value
3 attempts
Why
Enough to ride out a transient failure, short enough that a permanent one is reported the same day.
Maximum attempt budget
Value
8
Why
A requested budget is clamped server-side rather than rejected, so an exaggerated value cannot create an unbounded retry loop.
Backoff
Value
5s → 20s → 80s → …, factor 4, capped at 5 min
Why
Growth fast enough to stop hammering a service that is down, capped so a long queue still drains.
Attempt accounting
Value
Incremented when the attempt starts
Why
A worker that dies mid-attempt does not get a free retry; the attempt it was running is visible in the row.
Explicit retry
Value
FAILED → PENDING with a fresh budget
Why
An operator who fixed the cause re-queues the job; anything not in FAILED answers not_retryable.
Terminal retention
Value
30 days by default, configurable
Why
Finished rows are purged by a scheduled job; nothing that is still PENDING, RUNNING or RETRY_WAIT is ever eligible.

Bounds and refusals

Every way a task call can be refused
Condition	Answer	Detail
Payload or result over the cap	`400` `payload_too_large`	Each is capped at 64 KiB and must be valid JSON. The message names which side exceeded the cap.
Unknown kind	`400` `unknown_task_kind`	The dispatcher works on a closed set of four kinds.
Cancelling a finished job	`400` `not_cancellable`	A SUCCEEDED, FAILED or CANCELLED job can only be retried (if FAILED) or ignored.
Cancelling a running job	`400` `running_not_cancellable`	The row is left in RUNNING, because that is the truth.
Retrying anything not FAILED	`400` `not_retryable`	Only a failed job has an attempt budget left to spend.
Reading or cancelling a foreign job	`404` `not_found`	Owner and id are both in the query, so the answer is the same as for a job that never existed.
Listing without bounds	`GET /api/tasks`	Pages with `limit` (1–200, default 50) and `offset`; an out-of-range value is clamped before it is narrowed to an int.

Payload or result over the cap
Answer
400 payload_too_large
Detail
Each is capped at 64 KiB and must be valid JSON. The message names which side exceeded the cap.
Unknown kind
Answer
400 unknown_task_kind
Detail
The dispatcher works on a closed set of four kinds.
Cancelling a finished job
Answer
400 not_cancellable
Detail
A SUCCEEDED, FAILED or CANCELLED job can only be retried (if FAILED) or ignored.
Cancelling a running job
Answer
400 running_not_cancellable
Detail
The row is left in RUNNING, because that is the truth.
Retrying anything not FAILED
Answer
400 not_retryable
Detail
Only a failed job has an attempt budget left to spend.
Reading or cancelling a foreign job
Answer
404 not_found
Detail
Owner and id are both in the query, so the answer is the same as for a job that never existed.
Listing without bounds
Answer
GET /api/tasks
Detail
Pages with limit (1–200, default 50) and offset; an out-of-range value is clamped before it is narrowed to an int.

Walkthrough

Submit, poll, then recover a failed job

201 → 200 → 200 OK

The same key is used for both submissions, so the second call is answered from the store. Only after the job reaches FAILED does the explicit retry become legal.

Request

Submit, poll, then recover a failed job

# 1 — submit (retry-safe)
POST /api/tasks
{
  "kind": "dlq.replay",
  "payload": { "ids": ["a1","a2"] },
  "idempotency_key": "replay-2026-10-19",
  "max_attempts": 3
}

# 2 — poll one job
GET /api/tasks/6f1c…

# 3 — it exhausted its attempts
GET /api/tasks/6f1c…
{ "task": { "status": "FAILED", "attempt": 3, "max_attempts": 3 } }

# 4 — retry it, now that the cause is fixed
POST /api/tasks/6f1c…/retry

Response

Response body

{
  "task": {
    "id": "6f1c…",
    "owner_user_id": 5094837833,
    "kind": "dlq.replay",
    "status": "PENDING",
    "attempt": 0,
    "max_attempts": 3,
    "created_at": 1790000000
  },
  "already_existed": false,
  "id": "6f1c…",
  "status": "PENDING"
}

A cancel that is refused, honestly

400 Bad Request

A reminder that the API reports state rather than intent: the row is not silently marked CANCELLED, so a poll series never shows a value the worker contradicts.