Operations

Bootstrap

notify.svc generate > .notify.yaml     # scaffold a boot config
# edit dbpools / default.dbpool / domain
notify.svc -f .notify.yaml

At boot the service loads the boot config, warms any tenants listed under loadtenants, initializes the gateway (which forces every provider to register, so misconfiguration surfaces at boot rather than at first send), mounts routes, and flips health/ready.

Schema is embedded and applied through the data-admin plane — generate a plan, then apply it (apply requires superadmin):

curl -X POST https://notify.internal/v2/admin/data/plans   -H "Authorization: Bearer $TOKEN" $H
curl -X POST https://notify.internal/v2/admin/data/plans/$PLAN_ID/apply   -H "Authorization: Bearer $SA_TOKEN" $H

Run flags: -f/--config (default .notify.yaml), -p/--port, --sid, --datacenter, --cluster, --config_url, --config_apikey, --servicediscovery_url, --servicediscovery_apikey, --svccert, --svckey, --rootcacert, --jwt_public_key, --zerotrust. Subcommands: version, generate.

Retry and dead-letter

Async sends are swept by two per-tenant background jobs — one for pending items (daemon.pendingdelay, default 10s), one for retries (daemon.faileddelay, default 100s).

Status machine: pending → sending → sent | failed, and failed → dead once the retry budget is exhausted.

Behavior	Detail
Retry budget	max_attempts, default 5
Backoff	30s × 2^(n-1), capped at 1 hour — no jitter, so a provider outage produces synchronized retry waves
Rate-limited items	Do not consume a retry attempt; rescheduled at now + Retry-After + 2s
Dead letter	Terminal dead status on the delivery record, with the error code/message and per-attempt history retained
Scheduled sends	Honored via sendat — the sweep skips items until their send time

There is no automatic dead-letter replay. To retry a dead item, inspect it and re-send with a fresh idempotency key.

Delivery inspection

Every attempt emits a structured notify.send log line carrying tenant, channel, provider, outcome, template, recipient hash, duration, idempotency key, and any error. Recipient values are hashed, not logged in clear.

Durable state lives in two tables, indexed for exactly these queries:

• Delivery records — query by status (find everything failed/dead) or by idempotency_key (trace one send).
• Outbox rows — join by outbox id for the original request payload.

Rate limiting

Two independent systems:

Inbound (HTTP) — per-class limits on the API surface. Only classes you configure are enforced; there are no implicit defaults.

Outbound (per provider) — token buckets derived from each provider’s declared capability, overridable per tenant. Bucket scope is one of global, per-tenant, per-sender, or per-from-number. When every provider for a channel is limited, the send returns rate_limited with the largest Retry-After across the denying windows. A rate-limit store error is treated as a denial with a conservative 30s backoff.

Both share the store configured at notify.ratelimit_store (memory, distrib, or tiered).

Failure modes

Symptom	Cause / action
Sends succeed but ignore limits	The rate limiter failed to construct — the gateway runs unlimited and logs an error. Alert on this
Inbound limits not applied	Inbound limiter construction failed; traffic is allowed through
Unexpected provider chosen	No routing rules for the channel → falls back to the first registered provider
notify-route-all-providers-failed	Every provider failed; the error data lists providers tried and the last error
Duplicate messages after a restart	Idempotency and suppression are in-process memory — see the caveat below
Queue not draining for one tenant	Tenant not resolvable / meta client missing → the sweep is skipped for that tenant
/notify/list reports failures	Per-member failures don’t abort the run; check dispatched vs failed

Single-replica assumptions

Idempotency and suppression state are currently held in process memory, not in the database. Running more than one replica will double-send on retries and lose suppression across restarts. The suppression, provider_receipt, and notification_audit tables exist and are indexed, but nothing writes them yet — audit is log-only today.

Known gaps

Worth knowing before you design around them:

• Delivery receipts / bounce webhooks — the provider callback endpoints are implemented in the framework but not mounted by the service, so bounce and delivery-status callbacks are not received.
• Secret references in provider config are documented but not yet resolved — provider config values are read as-is.
• Quiet hours and sandbox capture are designed but have no runtime path.
• Health is a flat liveness/readiness flip — it does not check provider reachability or store health.
• Metrics are specified but not yet emitted; use the notify.send log line for now.
• Circuit breakers per provider are not implemented — a hard-down provider is retried on every attempt until the budget is exhausted.