Documentation menu

Getting started

New in v2.0

Feature deep dives

System atlas

Reference

Learn

Operations

Operations & troubleshooting

Runbook for the stack: read the health board, query attributable log history, handle FloodWait, resolve database contention, replay dead letters and roll back.

Verified against the checkout built for v2.2.4 · updated 20 Oct 2026

On this page

Health checks
Log history
FloodWait
Database contention
Dead-letter queue
Backup & rollback
Upgrades

This is the page you open when something is wrong. Every section ends in a command or an endpoint you can actually run, and the log signals below are the vocabulary the stack logs in.

Health checks

Start broad, then narrow. The CLI status board answers "is anything red", the readiness endpoint answers "is this deployment configured", and doctor answers "why".

Three levels of diagnosis

atf status             # live board: ports, reachability, task counts
atf validate --strict  # config, compose, catalog parity, secrets scan
atf doctor --fix       # deep checks across DB, binaries, ports, gRPC

# The same, over HTTP
curl -s localhost:8080/api/health/services | jq
curl -s localhost:8080/api/system/setup | jq '.ready, .env'

Signal	Means
`ready: false`	At least one required environment key is missing or a service is unreachable.
`core: false`	The Python core is down — nothing is being read from Telegram.
`logger: false`	Log endpoints answer 502; delivery continues but history is not being recorded.
`api: false`	Clients cannot reach the control plane.
`web: false`	Only the dashboard is affected; the bot and API still work.

Log history

History is attributable: entries name the actor, the service, the level, the subject task and the correlation id. That is what lets you answer "who changed this, and what happened next" hours later.

Query history from the CLI

atf logs tail --follow -n 50     # live stream
atf logs query --level ERROR     # filtered query
atf logs stats                   # counts by service and level
atf logs export --format csv --out logs.csv

Query history over HTTP

# Bounded paging, tenant-scoped server-side
curl -s "localhost:8080/api/logs?service=core&level=ERROR&limit=50" \
  -H "Authorization: Bearer $TOKEN" | jq '.items'

# Aggregates, one task's history, and export
curl -s localhost:8080/api/logs/stats -H "Authorization: Bearer $TOKEN" | jq
curl -s localhost:8080/api/rules/<id>/logs -H "Authorization: Bearer $TOKEN" | jq
curl -s localhost:8080/api/logs/export -H "Authorization: Bearer $TOKEN" -o logs.json

Pipeline log vocabulary
Signal	Meaning	What to do
[PIPELINE:ROUTE]	A task matched this source message.	Normal. Check the task list if you expected a different target.
[PIPELINE:FILTER_DROP]	A filter rejected the message.	Expected only if the rule says so — inspect the task policy.
[PIPELINE:CLEAN]	Cleaner removed content.	Normal when cleaning is enabled.
[PIPELINE:REWRITE]	The AI pass rewrote the text.	If it did not run, check the task toggle and provider reachability.
[PIPELINE:RESTRICTED_BYPASS]	Protected content was re-uploaded instead of forwarded.	Normal copy-protection handling.
[PIPELINE:QUEUE]	The job was persisted with a lease.	A job stuck here means workers are not draining.
[PIPELINE:DELIVERED]	Telegram accepted the delivery.	Done.
[PIPELINE:RETRY_WAIT]	The job is waiting out its backoff.	A paused task keeps jobs here by design.
[PIPELINE:DLQ]	Retries were exhausted.	Read the recorded error, fix the cause, then replay.

[PIPELINE:ROUTE]
Meaning
A task matched this source message.
What to do
Normal. Check the task list if you expected a different target.
[PIPELINE:FILTER_DROP]
Meaning
A filter rejected the message.
What to do
Expected only if the rule says so — inspect the task policy.
[PIPELINE:CLEAN]
Meaning
Cleaner removed content.
What to do
Normal when cleaning is enabled.
[PIPELINE:REWRITE]
Meaning
The AI pass rewrote the text.
What to do
If it did not run, check the task toggle and provider reachability.
[PIPELINE:RESTRICTED_BYPASS]
Meaning
Protected content was re-uploaded instead of forwarded.
What to do
Normal copy-protection handling.
[PIPELINE:QUEUE]
Meaning
The job was persisted with a lease.
What to do
A job stuck here means workers are not draining.
[PIPELINE:DELIVERED]
Meaning
Telegram accepted the delivery.
What to do
Done.
[PIPELINE:RETRY_WAIT]
Meaning
The job is waiting out its backoff.
What to do
A paused task keeps jobs here by design.
[PIPELINE:DLQ]
Meaning
Retries were exhausted.
What to do
Read the recorded error, fix the cause, then replay.

FloodWait

Telegram rate-limits accounts, and fighting it is how accounts get banned. ATF treats a FloodWait as a scheduling signal: the account is marked time-limited, its deliveries pause, jobs stay queued, and everything resumes when the window closes.

The client pool records the affected account and the moment it becomes usable again.
Workers skip that account and keep serving the others — one limited account does not stall the installation.
When the window expires the queue drains automatically. Repeated FloodWaits mean your rates are too high, not that the code is broken.

Database contention

Under a burst of concurrent tasks, SQLite can report database is locked if writes pile up. The fix is either a better journal mode or a real database engine.

Keep SQLite, make it concurrent

sqlite3 data/atf.db "PRAGMA journal_mode=WAL; PRAGMA busy_timeout=5000;"
atf db status          # integrity and lock visibility
atf db snapshot        # consistent copy before changing anything

Move to PostgreSQL

# .env
DATABASE_URL=postgresql://atf_user:password@localhost:5432/atf_db

atf db snapshot
atf system services    # restart the stack
atf doctor             # confirm every service reconnected

Dead-letter queue

A job that exhausts its retries stops guessing and lands in the DLQ with the error that killed it. Replay is deliberate: fix the cause, then replay, one job or all of them.

Inspect and replay

atf data queue            # depth and in-flight jobs
atf data dlq              # list dead-lettered jobs with errors
atf data dlq --replay     # replay once the cause is fixed

The same over HTTP

curl -s localhost:8080/api/queue -H "Authorization: Bearer $TOKEN" | jq
curl -s localhost:8080/api/dlq -H "Authorization: Bearer $TOKEN" | jq '.items'
curl -s -X POST localhost:8080/api/dlq/<id>/retry \
  -H "Authorization: Bearer $TOKEN" | jq

Backup & rollback

Encrypted export / import

atf data backup-export --out atf-$(date +%F).atf
atf db snapshot --out data-snapshot.db
atf sessions backup --out sessions.bak

# Restore on this host or another
atf data backup-import --in atf-2026-10-16.atf

Roll back an upgrade

# Pin the previous tag, rebuild what changed
git fetch --tags
git checkout v1.5.2
docker compose build && docker compose up -d

atf validate --strict
curl -f localhost:8080/healthz

Upgrades

The gated path

atf validate --strict    # refuse to start from a broken state
atf system upgrade       # pull, rebuild, restart, verify
atf status --watch       # keep an eye on the board afterwards
atf release verify       # manifest + portable build integrity

Release-by-release notes live on the releases page; the command surface is documented in the CLI reference.