Troubleshooting runbook

Fast triage

Four commands, cheapest first, tell you whether the failure is the stack, the configuration or the rules. Run them before reading any single section below.

The same questions are answerable over HTTP, which is what a dashboard or a monitoring script uses:

curl -sf localhost:8080/healthz                       # is the gateway alive
curl -s localhost:8080/api/health/services | jq .status  # healthy | degraded
curl -s localhost:8080/api/system/setup | jq '.ready, .missing_required'
curl -s localhost:8080/api/system/env | jq '.ok, .missing_required'

A service will not start, or is unhealthy

Symptom. The dashboard is blank, forwarding stopped, or a service shows red on the board.Cause. One container is down — a crash loop, a missing env key, or a port already bound. The stack tolerates a down web or logger without stopping delivery, but a down core reads nothing from Telegram.

docker compose ps                 # state and ports of every container
atf docker ps                     # the same, ATF containers only
atf docker logs core              # docker logs for atf-core (tail 100)
atf system services               # state, health, endpoint and version; exit 1 if any is down
atf system services --json        # {services:[...], healthy:N, total:4, failing:[...]}

Confirm the fix with atf system services (all four OK) andcurl -s localhost:8080/api/health/services | jq .status returning healthy. The port contract each service is probed on is fixed:

Dashboard login fails or loops

Symptom. The link from the bot bounces back to the login screen, or the dashboard keeps asking you to re-authenticate. Cause. A dashboard link carries a single-use code, not a session token: codes live 600 seconds, are burned on first exchange, and only the core that minted one can trade it for a session. A second tab, a refresh or a link replayed from chat history all fail the same way.

# Exchange a bot-issued code for a 7-day session
curl -s -X POST localhost:8080/api/auth/exchange-bot-token \
  -H 'Content-Type: application/json' \
  -d '{"login_code":"<code from the bot>"}' | jq

# Ask for a fresh one-tap link (needs a Bearer token)
curl -si localhost:8080/api/auth/bot-login-link \
  -H "Authorization: Bearer <token>" | grep -i retry-after

atf logs query --service core -s LOGIN-CODE -n 20

[LOGIN-CODE] refused reason=already_used user_id=1001
[LOGIN-CODE] refused reason=expired      user_id=1001
[LOGIN-CODE] cooldown user_id=1001

A task creates but never forwards

Symptom. The rule saves and shows as active, but posts from the source never reach the target.Cause. Almost always one of five things: the bot is not an admin in the source or the target, the channel id is a username where a numeric id is required (or the reverse), the rule is paused, deduplication recognises the post, or a filter drops it.

# Simulate a post through the live rule engine: no delivery is made
atf rules simulate <task-id-or-name> --text "the post text" --show

# The answer names matched / would_forward / reason / transformed_text,
# so a filter or a pause is visible instead of guessed.
atf rules list --json | jq '.[] | {id, name, enabled, paused}'

Messages stuck in the queue or dead-lettered

Symptom. Posts are accepted but nothing arrives, or the dashboard shows a growing backlog.Cause. A job is waiting in the durable queue or has exhausted its retries and landed in the dead-letter queue. Replay is deliberate: fix the cause first, then replay.

atf data queue                 # depth and in-flight jobs by state
atf data dlq                   # dead-lettered jobs with their errors
atf data dlq retry <id>        # replay one job after the cause is fixed
atf data dlq purge             # clear the DLQ once nothing is worth replaying

curl -s localhost:8080/api/queue -H "Authorization: Bearer ***" | jq
curl -s localhost:8080/api/dlq -H "Authorization: Bearer ***" | jq '.items'
curl -s -X POST localhost:8080/api/dlq/<id>/retry \
  -H "Authorization: Bearer ***" | jq

FloodWait & protected posts

Symptom. Deliveries stop for a while, then resume on their own; or a post arrives re-uploaded instead of as a forward. Cause. Telegram rate-limits accounts (FloodWait), and channels can forbid forwarding. Neither is a bug.

1. The account is marked time-limited for the wait Telegram named, and its deliveries pause.
2. Jobs stay queued in RETRY_WAIT; other accounts keep serving, so one limited account never stalls the installation.
3. When a forward is refused because the channel is protected, the post is re-uploaded instead. Repeated FloodWaits mean your rates are too high, not that the code is broken.

“Port already in use”

Symptom. A service fails to start with port already in use, often 8088or 8080. Cause. Another process — usually a previous stack that was not shut down — owns the port. The contract is fixed: api 8080, web 8088, core 50051, logger 50052.

atf validate --runtime      # the ports check names the owning process
atf system restart          # recreate the stack on the current .env
docker compose down         # or tear the old stack down first
atf doctor                  # confirm 8080, 8088, 50051, 50052 all answering

429 rate limiting

Symptom. The dashboard or a script gets 429 Too Many Requests. Cause.The API enforces per-route limits in front of the control plane, and the unauthenticated auth endpoints are limited hardest. Every refusal names itself, so a client can pace itself instead of discovering the limit by being refused.

curl -si localhost:8080/api/rules -H "Authorization: Bearer ***" | grep -i 'x-ratelimit\|retry-after'

X-RateLimit-Limit: 120
X-RateLimit-Remaining: 0
Retry-After: 12
X-RateLimit-Degraded: 1     # limit enforced per process (Redis unreachable)

An AI provider is unreachable

Symptom. Rewritten posts arrive unmodified, or a rule with a rewrite configured stops sending.Cause. The provider is down, the key is wrong, or the rewrite was turned off for that task. A provider failure must never block delivery: the outcome is decided by the rule's fallback policy.

# Probe a provider key without sending anything
curl -s -X POST localhost:8080/api/ai-configs/check-key \
  -H "Authorization: Bearer ***" -H 'Content-Type: application/json' \
  -d '{"provider":"openai","model":"gpt-4o-mini"}' | jq '.success, .healthy, .error_code'

# Confirm the per-task toggle in the logs
atf logs query --level WARN -s AI_REWRITE -n 20

# The pipeline asks one question: ai_rewrite_active.
# A task that has an ai_config_id rewrites, unless the task panel's
# paid-addon switch vetoes it. That switch is paid_addons.ai_rewrite.
atf rules settings set <task> paid_addons.ai_rewrite=false
atf rules simulate <task> --text "sample" --show

Log history is empty or truncated

Symptom. Old entries are gone, or history is thinner than expected. Cause.History is bounded on purpose. The logger keeps a bounded in-memory queue and, when it is full, evicts theoldest queued record (reported as dropped) so the newest context survives. Age-based retention then deletes rows past the policy age — seven days by default — while audit and security categories are excluded from deletion by policy.

atf logs stats                        # volume by service and level
atf logs query --level ERROR --hours 24
atf logs purge --older-than-days 30   # a separate, audited action — not the same as shedding
curl -s localhost:8080/api/logs/policy -H "Authorization: Bearer ***" | jq

A container is older than your .env

Symptom. The bot reports it is unconfigured although .env is filled, or the validator warns container-env. Cause. Environment variables are read when a container is created, so a container started before the last .env change keeps the old values.

atf validate --strict      # container-env flags a container missing a key .env sets
atf system restart         # recreate the containers on the current .env
atf doctor                 # confirm every service reconnected

Collect evidence before you ask for help

A bug report is only actionable with the correlation id, the log slice and a machine-readable doctor report. Gather all three; they answer most follow-up questions before they are asked.

# 1. Correlation id — every request carries one end to end
atf logs query --level ERROR -n 20          # copy the correlation id
atf logs query --corr <correlation-id>      # the whole story for that request

# 2. A log export (JSON or CSV)
atf logs export --format json -o atf-logs.json --hours 24

# 3. Machine-readable health
atf doctor --json              # exit 0 clean, 1 if any check failed
atf validate --json            # one JSON object on stdout
atf system services --json     # {services:[...], healthy:N, total:4, failing:[...]}