Operations

Audience: DevSecOps — engineers deploying audit.svc. Single-node covers everything we support today; multi-replica is constrained by lib-audit’s single-writer-per-log file lock and is not yet documented (see §11).

1. Prerequisites

Requirement	Detail
OS	Linux x86_64 / arm64. macOS supported for dev only.
Privileged user	Run as audit (uid/gid created by package). Do not run as root.
Filesystem	A persistent local directory for tenant logs (default /var/lib/audit/logs). Must survive process restart; NOT a tmpfs; ext4/xfs recommended. POSIX file locks (flock) must be supported — NFS/CIFS may not honor them correctly.
Network	One TCP port (default :8443). Outbound HTTPS to vault, the chassis config service, and your IAM/JWT issuer.
TLS material	Provided by the kis.ai chassis (vault-issued service cert). Do not embed certs in .audit-svc.yaml.
Vault	A reachable kis.ai vault with per-tenant Ed25519 keys at the configured vault_path_template. See §4.
Config service	Reachable kis.ai chassis config service (config.svc). The chassis client is auto-wired by svcboot.
Time	NTP-synced. Tree-head timestamps and JWT nbf/exp rely on accurate wall clock.

2. Install

2.1 Build from source

There are no released tarballs yet. Build from a source checkout of audit.svc:

cd path/to/audit.svc
go build -trimpath -tags netgo -o dist/audit-svc.bin .

Or use the chassis build.yaml driver (matches CI):

metis run build.yaml build       # produces dist/audit-svc.bin

2.2 systemd unit

/etc/systemd/system/audit.svc.service

[Unit]
Description=audit.svc — tamper-evident audit logging
After=network.target

[Service]
Type=simple
User=audit
Group=audit
WorkingDirectory=/etc/audit
ExecStart=/opt/audit/audit-svc.bin -config /etc/audit/audit-svc.yaml
Restart=on-failure
RestartSec=2
LimitNOFILE=65536

# Hardening
ProtectSystem=strict
ProtectHome=true
ReadWritePaths=/var/lib/audit /var/log/audit
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

sudo install -d -o root -g root /opt/audit
sudo install -m 0755 dist/audit-svc.bin /opt/audit/
sudo install -d -o root -g audit -m 0750 /etc/audit
sudo install -d -o audit -g audit -m 0700 /var/lib/audit/logs

sudo systemctl daemon-reload
sudo systemctl enable --now audit.svc
sudo systemctl status audit.svc

3. Bootstrap config

The full schema is in the API reference, and an annotated example config is provided as .audit-svc.yaml. Minimum production shape:

name: audit
port: 8443

# vault.url + ancillary auth keys are read by lib-vault.
# These come from the chassis cluster config; you usually
# do NOT inline them here.
vault:
  url: "https://vault.kis.ai:8200"

audit:
  base_dir: "/var/lib/audit/logs"

  signer:
    vault_path_template: "audit/{tenant}/ed25519-private-key"
    key_id_template:     "{tenant}-key"

  routing:
    - log_id_pattern: "prod/*"
      durability: "strict"               # fsync per Append
      seal_interval: "60s"
      segment_max_records: 100000
      segment_max_bytes: 67108864        # 64 MiB
    - log_id_pattern: "**"
      durability: "batched"
      seal_interval: "300s"
      segment_max_records: 100000
      segment_max_bytes: 67108864

  sync:
    - log_pattern: "**"
      targets:
        - kind: "fs"
          path: "/var/backup/audit"
          cadence: "60s"

  # anchor: ...   # all kinds are stubbed in lib-audit v0.1

durability options:

Mode	Behavior	Use when
strict	fsync per Append — zero loss on crash.	Production / compliance-critical logs.
batched	Periodic fsync.	High-throughput logs where seconds of loss are acceptable.
async	Relies on OS page cache.	Dev / ephemeral logs only.

4. Vault — per-tenant signing keys

audit.svc fetches each tenant’s Ed25519 key from vault at the templated path. Provision before the tenant submits.

4.1 Generate an Ed25519 key

openssl genpkey -algorithm ed25519 -out /tmp/cust-acme.pem
# Inspect: openssl pkey -in /tmp/cust-acme.pem -text -noout

The PEM should have either -----BEGIN PRIVATE KEY----- (PKCS#8, the default openssl format) or -----BEGIN ED25519 PRIVATE KEY----- (the raw-bytes format lib-audit’s local signer writes). Both are accepted by signer.VaultFactory.

4.2 Store in vault at the templated path

If vault_path_template is audit/{tenant}/ed25519-private-key, the path for tenant cust-acme is audit/cust-acme/ed25519-private-key:

vault kv put audit/cust-acme/ed25519-private-key value=@/tmp/cust-acme.pem

(Adjust to your vault auth + mount conventions. The chassis lib-vault GetSecret(ctx, name) is what does the lookup; it returns the secret string as-is.)

4.3 Smoke-test

After provisioning, the first /submit for that tenant will fetch the key. Watch the log:

INFO  audit: created new log  tenant=cust-acme log_id=auth root=/var/lib/audit/logs/cust-acme/auth

If you instead see:

ERROR vault signer: get secret "audit/cust-acme/..." for tenant "cust-acme": ...

the path or vault policy is wrong. Fix and re-submit.

4.4 Key rotation

Today there is no live invalidation — the in-memory cache holds the key until the process restarts. To rotate:

1. Put the new key at the same vault path.
2. Restart audit.svc (systemctl restart audit.svc).
3. The next /submit for that tenant fetches the new key.

A live Invalidate(tenant) hook is wired in signer.VaultFactory but not yet exposed; it is tracked in the internal design backlog under “Signing / vault — Key rotation.”

5. Storage planning

Knob	Default	Effect
segment_max_records	100 000	Segment rolls after this many leaves.
segment_max_bytes	64 MiB	Segment rolls after this many bytes (whichever comes first).
tile_height (lib-audit Options)	8	Merkle tile density — affects on-disk overhead, not configurable from our YAML yet.

Disk footprint per log:

• One log.json (manifest) + one manifest.json per log dir.
• segments/NNNN.seg files — append-only, rotated by the knobs above.
• tiles/L<level>/<idx>.tile files — Merkle tree internals (small).
• heads/<tree_size>.head files — one per /seal call. Long-running logs accumulate many; they’re tiny but ls becomes slow if your filesystem dislikes large directories.
• keys/*.pub — public-key cache for verification (optional in our current setup, but the directory exists).

Rough rule: budget ~1.2× the sum of submitted payload bytes for a log’s disk footprint, plus the per-segment fixed overhead.

6. Health probes

GET /health   → 200 ok      # liveness only — process is up
GET /ready    → 200 ok      # readiness — chassis bootstrap complete

Both unauthenticated; no body required.

Kubernetes:

livenessProbe:
  httpGet: { path: /health, port: 8443, scheme: HTTPS }
  initialDelaySeconds: 10
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /ready,  port: 8443, scheme: HTTPS }
  initialDelaySeconds: 5
  periodSeconds: 5

Neither probe checks vault, disk, or worker health. That gap is tracked in the internal design backlog under “Health / observability.”

7. Logs to watch

Service uses lib-zero/logger; output goes to stdout as JSON.

Expected (info):

audit: opened log              tenant=... log_id=... tree_size=...
audit: created new log         tenant=... log_id=... root=...
audit.sealer: sealed           tenant=... log_id=... tree_size=...   # debug-level

Warn — investigate but service is still working:

audit.sealer:  seal failed; will retry with backoff   failures=N
audit.syncer:  failed; will retry with backoff        target=...
audit.syncer:  unknown target kind                   kind=...   # config error

Error — service degraded:

audit: resolve vault client                  # PreStart failed → process exits
audit.worker: tick panic recovered           # bug; capture stack and file
audit.worker: panic recovered; restarting after pause  # outer-loop panic

For each failures=N, the next attempt is at 30s × 2^(N-1), capped at 1 hour. A target with failures=10 is parked for an hour before the next try. To force a retry sooner, restart the service (the pool resets backoff state on Start).

8. Backup

The on-disk log directory (audit.base_dir) is the source of truth. Back it up cold (snapshot + tar) or with rsync — append-only segments play well with incremental backups.

Sealing is NOT required before backup. The Merkle tree state can be reconstructed from segments on audit.Open. If you back up mid-write you may capture a partially-written tail record; audit.Open recovery truncates that on next open.

The configured audit.sync worker also ships sealed segments to your chosen target (fs works today; s3 / gcs / azure are stubbed in lib-audit). Use it as a continuous secondary, not a backup replacement.

9. Common operational scenarios

Vault outage

• Symptom: /submit for any new tenant returns 500 with vault signer: get secret ...: connection refused. Tenants whose signers are already cached continue to work.
• Action: restore vault. No restart required; in-flight failures will succeed once vault is back.

Disk full

• Symptom: /submit returns 500 with no space left on device from the segment writer.
• Action: free space (rotate old logs externally if possible; the service does not delete its own data) and retry.

Tenant directory needs to be restored from backup

• Stop audit.svc.
• Restore the entire <base_dir>/<tenant>/ tree from backup. Keep permissions: audit:audit, mode 0700 / files 0600.
• Start audit.svc. The next request for any log under that tenant re-opens via audit.Open — sequence numbers and Merkle state are reconstructed.

”Log is locked” error on /submit

lib-audit’s POSIX file lock is held by a still-running process. If no audit.svc is running, the lock is stale (process killed without cleanup); delete <log_dir>/.lock and retry. If audit.svc is running, you have two replicas pointing at the same log directory — see §11.

Pre-warm logs at startup

logmgr opens logs lazily on first request. For a deployment where you want known logs open before the first user request (e.g. so workers start sealing immediately), submit a no-op request for each expected log_id after restart, or wait for the first real request.

10. Worker observability

The three workers (sealer, syncer, anchorer) log on debug/info; the per-target backoff counter (failures=N) is your primary signal that a target is sick.

There is no built-in /admin/workers debug endpoint yet — that’s tracked in the internal design backlog under “Health / observability.” For now, grep the service log for failures=.

11. Multi-replica — known constraint

lib-audit enforces single-writer-per-log with a POSIX file lock (flock(LOCK_EX|LOCK_NB)). Two audit.svc replicas pointing at the same (tenant, log_id) on shared storage will see the second one fail to open with ErrLogLocked.

To safely run more than one replica, you need either:

• Affinity routing — load-balance so all requests for a given (tenant, log_id) hit the same replica. Sticky on a hash of (tenant, log_id). Requires care in the gateway/ingress layer.
• Shared no-write storage with one writer — designate one replica as the writer for each shard.

We don’t ship either pattern today. Run single-replica until this work lands.

12. Upgrade

audit.svc is stateless beyond the on-disk logs. Standard rolling upgrade:

1. Stop the running instance.
2. Replace /opt/audit/audit-svc.bin.
3. Start. The chassis bootstrap re-reads config; logmgr re-opens logs lazily on first request.

On-disk segment format has not changed in v1.0 and the upstream lib-audit roadmap calls out any future breaking change explicitly. Read the upstream changelog before upgrading the lib-audit pin.