Operations

Build

go build -o dist/edge-intake .

Or via the fleet-uniform build.yaml (clean / build / test); the build task stamps version.Version, .BuildCommit, and the build timestamp through -ldflags. Intake version then reads like every other kis.ai binary:

Intake (intake) - v0.2.0
Commit Hash:…
Build Timestamp:2026-07-10 03:22:10 UTC
Built 1 secs ago

DuckDB is linked via CGO (github.com/duckdb/duckdb-go/v2), so a C toolchain is required at build time. The binary itself is standalone.

Run

Startable from a config file, from flags alone, or both:

# config file
edge-intake --config /etc/edge-intake/config.yaml

# flags only — no file
Intake \
  --http-bind 127.0.0.1:8081 \
  --storage-backend duckdb --wal-jsonl \
  --duckdb-path /var/lib/edge-intake/intake.duckdb \
  --jsonl-dir  /var/lib/edge-intake/data \
  --forms-dir  /etc/edge-intake/forms \
  --log-output file --log-file /var/log/edge-intake/edge-intake.log

Subcommands:

command	purpose
edge-intake (no subcommand)	run the server
Intake version	print version + build metadata
Intake config generate	print a default config.yaml to stdout
Intake config show	print the effective config after file + flag resolution (secret redacted)

Configuration

Resolution order (highest wins): CLI flags → config file → built-in defaults. Only flags explicitly set on the command line override the file (implemented with pflag Changed(), the analogue of stdlib flag.Visit) — an unset flag never clobbers a config value with its zero default. Every config key has a matching --dashed-flag.

http:
  bind: 127.0.0.1:8081     # loopback/private only — never internet-reachable
  max_body_bytes: 65536    # 64 KB cap before parse -> 413
  max_fields: 100
  max_parts: 100
  read_timeout: 5s
  write_timeout: 5s
  idle_timeout: 60s

storage:
  backend: duckdb          # duckdb | jsonl
  wal_jsonl: true          # JSONL write-ahead log under DuckDB
  batch_max: 64            # group-commit batch ceiling
  batch_wait: 25ms         # group-commit linger
  queue_depth: 1024        # bounded writer channel -> 503 on overflow

duckdb:
  path: /var/lib/edge-intake/intake.duckdb
jsonl:
  dir: /var/lib/edge-intake/data
  rotate: daily            # daily | none

forms:
  dir: /etc/edge-intake/forms
  reload: startup          # startup | sighup

logging:
  output: file             # file | stdout | stderr
  format: json             # json | text
  level: info
  file: { path: /var/log/edge-intake/edge-intake.log, max_size_mb: 100, max_backups: 14, max_age_days: 30, compress: true }

ratelimit:
  per_ip: 5
  per_ip_burst: 20
  global: 200
  global_burst: 400
  ip_ttl: 10m
  max_ips: 100000

security:
  client_ip_header: X-Kis-Client-IP
  bind_gateway_only: true
  gateway_secret: ""       # prefer --gateway-secret / EDGE_INTAKE_GATEWAY_SECRET
  gateway_secret_header: X-Kis-Gateway-Secret

cors:
  origins: []              # e.g. ["https://www.example.com"]

Secrets are never inlined on a public box: security.gateway_secret is taken from --gateway-secret or the EDGE_INTAKE_GATEWAY_SECRET env var. config show redacts it.

Common flags: --config, --http-bind, --max-body-bytes, --storage-backend, --wal-jsonl, --duckdb-path, --jsonl-dir, --forms-dir, --log-output, --log-format, --log-file, --log-level, --ratelimit-per-ip, --ratelimit-per-ip-burst, --ratelimit-global, --client-ip-header, --gateway-secret, --notify-enabled, --email-provider, --email-from, --smtp-host, --smtp-port, --smtp-auth, --smtp-username, --teams-webhook.

Notifications: transport & operations

Per-form destinations live in the form YAML; this notify: section holds the shared transport and the master switch. Email has a provider — smtp (generic) or sendgrid (Web API v3).

notify:
  enabled: false           # master switch — off = no notifications
  workers: 4               # delivery worker pool
  queue_depth: 1024        # bounded; a full queue drops (counted), never blocks the 200
  max_retries: 4           # per notification
  retry_backoff: 2s        # exponential, capped at 30s
  timeout: 10s             # per attempt (dial + send)
  email:
    provider: smtp         # smtp | sendgrid
    from: "kis.ai Leads <[email protected]>"
    smtp:
      host: ""             # empty disables SMTP
      port: 587
      starttls: true
      auth: plain          # plain | xoauth2 | none
      username: ""
      password: ""         # auth=plain; prefer EDGE_INTAKE_SMTP_PASSWORD
      oauth:               # auth=xoauth2
        token_url: ""
        client_id: ""
        client_secret: ""  # prefer EDGE_INTAKE_SMTP_OAUTH_SECRET
        scope: ""
    sendgrid:
      api_key: ""          # prefer EDGE_INTAKE_SENDGRID_API_KEY
  teams:
    default_webhook: ""    # fallback for forms without their own; prefer EDGE_INTAKE_TEAMS_WEBHOOK

Provider recipes

Microsoft 365 via OAuth2 (recommended — Basic auth is being retired). Register an Azure AD app, grant it SMTP.Send, and use the client-credentials flow:

email:
  provider: smtp
  from: "kis.ai Leads <[email protected]>"   # must be a mailbox the app is allowed to send as
  smtp:
    host: smtp.office365.com
    port: 587
    starttls: true
    auth: xoauth2
    username: [email protected]               # the mailbox being sent from
    oauth:
      token_url: https://login.microsoftonline.com/<tenant-id>/oauth2/v2.0/token
      client_id: <azure-app-client-id>
      # client_secret via EDGE_INTAKE_SMTP_OAUTH_SECRET
      scope: https://outlook.office365.com/.default

Microsoft 365 via Basic auth (works until end of 2026, then disabled): same as above but auth: plain, username/password (the mailbox + its password), no oauth. Requires SMTP AUTH enabled for the mailbox and Basic auth not blocked by Security Defaults.

SendGrid — Web API v3 (no SMTP):

email:
  provider: sendgrid
  from: "kis.ai Leads <[email protected]>"    # a Verified Sender / authenticated domain
  sendgrid:
    api_key: ""                          # via EDGE_INTAKE_SENDGRID_API_KEY

SendGrid — SMTP relay (alternative): provider: smtp, host: smtp.sendgrid.net, port: 587, auth: plain, username: apikey (literally), password = the API key (via EDGE_INTAKE_SMTP_PASSWORD).

Operational notes:

• The dispatcher starts only when enabled: true and at least one transport is usable (a configured email provider, a default webhook, or a form with its own webhook); otherwise it logs a warning and stays off. The startup log shows the resolved email kind, e.g. email=smtp/xoauth2 or email=sendgrid.
• Secrets come from the environment, never inline on a public box: EDGE_INTAKE_SMTP_PASSWORD, EDGE_INTAKE_SMTP_OAUTH_SECRET, EDGE_INTAKE_SENDGRID_API_KEY, EDGE_INTAKE_TEAMS_WEBHOOK. config show redacts all of them.
• The XOAUTH2 access token is fetched via client-credentials and cached until ~30 s before expiry (shared across workers).
• Delivery is decoupled from durability: the record is committed to the sink before it is queued for notification. A crash between the two loses at most the notification (re-deliverable from the store), never the lead. Delivery counters are on /metrics under notify and logged at shutdown.
• On shutdown the queue is drained within the shutdown window (best-effort).

from must be an address the provider is authorised to send as (SPF/DKIM/DMARC aligned; a verified sender for SendGrid; a SendAs-permitted mailbox for O365), or mail is rejected or spam-foldered regardless of Intake.

Deployment topology (behind the gateway)

Internet ──TLS──▶ gateway.svc (api.kis.ai)
                     │ terminates TLS, extracts client IP,
                     │ routes locally (loopback / private net)
                     ▼
               Intake  ──▶ DuckDB file (default)
               (plain HTTP,  └▶ JSONL log   (WAL + alt)
                loopback bind)

• Intake speaks plain HTTP on loopback / a private interface only. It is never directly internet-reachable — reachability is enforced by the bind address plus the gateway being the only ingress.
• The client IP arrives via a single trusted header (client_ip_header, default X-Kis-Client-IP) set by the gateway. Intake does not parse X-Forwarded-For — one trusted source, no XFF-chain ambiguity. A missing or unparseable value fails closed to a single shared rate-limit bucket.
• Optional defense-in-depth: set a gateway_secret; requests must then carry the matching gateway_secret_header (constant-time compared) or get 403.

Storage & durability

Default write path is JSONL-as-WAL under DuckDB. A single writer goroutine owns the DuckDB connection (it is not concurrency-safe) and drains a bounded channel, group-committing each batch:

1. append the batch to the JSONL log and fsync,
2. BEGIN … INSERT (prepared) … COMMIT to DuckDB,
3. only then ack each waiting handler.

The handler returns 200 only after the write is durable — no submission is lost on a crash after the response. Low volume → batch size 1 (latency ≈ one insert); bursts amortize into fewer commits. A full queue returns 503 rather than growing goroutines against the DB.

Modes: backend: duckdb (+ optional wal_jsonl), or backend: jsonl (append + fsync only). The append-only JSONL log is safe to tail concurrently and lets any store be rebuilt.

Hardening posture

surface	control
SQL injection	identifiers from YAML + strict regex; values bound-only; one prepared stmt per form compiled at load
Body size	MaxBytesReader before parse → 413
Field caps	max field count, max multipart parts, per-field max_length
Content-Type	allowlist only → 415
Encoding	reject invalid UTF-8 and control chars
Schema	strict unknown-field rejection; type/range/format/enum validation
Rate limiting	per-IP + global token buckets; janitor evicts idle IPs (bounded map)
Bots	honeypot → silent 200, discard
Concurrency	bounded writer channel; request read/write/idle timeouts
CORS	origin allowlist (never *; no credentials)
Headers	X-Content-Type-Options: nosniff; JSON responses
PII in logs	metadata only, never payload contents
Reachability	loopback/private bind; gateway-only ingress; optional shared secret
Shutdown	drain + flush/commit on SIGTERM — zero loss

Observability

• GET /health — liveness; 200 {"healthy":true,"version":…}. This is the fleet-canonical path the gateway health-checks (matches lib-chassis httputil.IsHealthy); /healthz is an alias.
• GET /ready — readiness; 200 ready:true while serving, 503 ready:false while draining on shutdown (matches lib-chassis httputil.IsReady); /readyz is an alias.
• GET /metrics — JSON counter snapshot: requests (accepted, validation_failed, unknown_form, too_large, unsupported_media_type, bad_request, rate_limited, shedding, bot, forbidden, internal) and, when notifications are on, notify (enqueued, dropped, email_sent, email_failed, teams_sent, teams_failed).
• Per-request log line (slog) carries request_id, form_id, source_ip, status, latency_ms, bytes, outcome — never the payload.
• Logging is standalone (no coupling to the usage or audit services); ship the file/stdout externally if desired.

Graceful shutdown & signals

• SIGINT / SIGTERM → stop accepting, let in-flight handlers finish (they block on the durable ack), then drain and close the sink. Zero-loss.
• SIGHUP → force a log-file rotation.

Downstream sync

The separate analytics-sync tool (DuckDB/JSONL → Postgres/ClickHouse) should tail the JSONL log, not open the live DuckDB file (single-writer lock). The monotonic ULID id is the incremental watermark; JSONL rotation gives natural checkpoint boundaries. Out of scope for this binary — see the design spec.

Troubleshooting

symptom	cause / fix
refuses to start, “destructive diff on …“	a form retypes/removes an existing column — migrate explicitly or revert the field type
refuses to start, “compile form …“	a bad identifier / duplicate field / enum without values in a form file
every POST /f/{id} → 404	forms dir empty or wrong; check the forms loaded startup log
all submissions share one rate bucket	client_ip_header missing/misconfigured at the gateway → IPs collapse to unknown
403 forbidden on every request	gateway_secret set but the gateway isn’t sending gateway_secret_header
cross-origin form blocked by browser	add the site origin to cors.origins
503 shedding under load	writer queue full (queue_depth) or global ceiling (ratelimit.global) hit — raise limits or check the sink
notifications never arrive	notify.enabled false, no transport configured (see the startup log), or the form has no notify: block
notify.email_failed climbing	SMTP host/port/auth/from wrong, or relay rejecting the sender — check the error log; leads are still stored
notify.dropped > 0	delivery can’t keep up — raise notify.workers / queue_depth, or the transport is down