Verifying Audit

Audience: developers integrating against audit.svc, and operators verifying a deployment. Two questions, answered separately: is this instance working? and can I prove this entry is genuine?

The second is the one that matters in a compliance review, and it is deliberately something you can answer without trusting us — the whole design exists so a third party can check the maths themselves. It is in §4.

1. What to verify, and when

When	Verify	Section
After a deploy or config change	The service is reachable and a round trip works	§2
While integrating	Your calls behave, against a local stack	§3
During an audit, or on any entry you must prove	The entry is in the log, and the log has not been rewritten	§4
Continuously	The API contract still holds	§5

2. Post-deploy smoke

Run from any host that can reach the service, with a JWT issued by IAM for an identity in the tenant under test.

export AUDIT_URL=https://audit.example.internal
export JWT=eyJhbGciOi...
export TENANT=cust-acme

Readiness.

curl -fsSL $AUDIT_URL/ready                                  # → ok

Submit an entry.

curl -fsSX POST $AUDIT_URL/submit \
  -H "Authorization: Bearer $JWT" -H "X-Tenant: $TENANT" \
  -H "Content-Type: application/json" \
  -d '{"submission":{"log_id":"smoke","payload":"aGVsbG8="}}' | jq

The receipt carries the sequence the entry was assigned. The log does not need creating first — it comes into existence on first submit.

Force a seal, so there is a signed head to verify against rather than waiting out the sealer’s interval.

curl -fsSX POST $AUDIT_URL/seal \
  -H "Authorization: Bearer $JWT" -H "X-Tenant: $TENANT" \
  -H "Content-Type: application/json" \
  -d '{"log_id":"smoke"}' | jq

Read it back.

curl -fsSX POST $AUDIT_URL/read \
  -H "Authorization: Bearer $JWT" -H "X-Tenant: $TENANT" \
  -H "Content-Type: application/json" \
  -d '{"log_id":"smoke","from_seq":0,"to_seq":0,"with_body":true}'

Four calls. If all four pass, the service, its vault signing key and its tenancy configuration are all correct — which is more than /ready tells you, because readiness does not touch vault.

Failure signals

Symptom	Likely cause
`401 invalid attempt detected by user`	The JWT’s tenant claim does not match `X-Tenant`
`401 missing tenant in request`	The `X-Tenant` header is absent
`400 invalid tenant`	The `X-Tenant` value is not registered in the config service
`500 vault signer: get secret ...`	No signing key provisioned for this tenant. See Operations
`404 logmgr: no routing rule matches log_id`	The `log_id` matched no routing rule. Add one, or a catch-all `**`

The first three are all tenancy, and they are the common case on a first deployment. Check that the JWT claim and the header agree before looking anywhere else.

3. A local stack

For integration work, run the service against a dev vault and a config service:

services:
  vault:
    image: hashicorp/vault:latest
    cap_add: [IPC_LOCK]
    environment:
      VAULT_DEV_ROOT_TOKEN_ID: dev-token
      VAULT_DEV_LISTEN_ADDRESS: 0.0.0.0:8200
    ports: ["8200:8200"]

  config:
    image: kisai/config.svc:latest
    # see the Config block's deployment guide

  audit:
    image: kisai/audit.svc:latest
    depends_on: [vault, config]
    volumes:
      - audit-data:/var/lib/audit/logs
      - ./.audit-svc.yaml:/etc/audit/audit-svc.yaml
    ports: ["8443:8443"]

volumes:
  audit-data:

Then:

Provision a signing key for the tenant you will test with: vault kv put audit/test-tenant/ed25519-private-key [email protected]
Mint a JWT for test-tenant against IAM.
Run the smoke sequence from §2 against https://localhost:8443.

A dev-mode vault holds its keys in memory, so restarting it makes every signature produced before the restart unverifiable. That is fine for integration work, and it is exactly why a dev vault does not belong in an environment whose logs you intend to keep.

4. Independent verification

This is the recipe an auditor, a compliance pipeline, or your own reconciliation job runs. It proves two things: that an entry is in the log at the sequence claimed, and that the log’s root at that point was signed by the tenant’s key.

You do not need access to the service’s disk, or to us, to do it. The proof is self-contained arithmetic over hashes you can recompute.

Get an inclusion proof. POST /inclusion-proof for the (sequence, tree_size) you are checking.
Get a signed head. POST /head, and confirm its tree_size is at least the one from step 1. Keep the root_hash, signature and key_id.
Reconstruct the leaf hash from the original payload, using the canonical leaf encoding in the HTTP API reference.
Walk the proof upward. Combine the leaf hash with each proof hash in turn and compare the computed root against the root_hash from step 2. They must be identical.
Verify the signature on that head against the tenant’s Ed25519 public key.

If step 4 matches, the entry is in the tree. If step 5 verifies, that tree was signed by the holder of the tenant’s key. Together they mean the entry cannot have been altered or removed without producing a different root — which the signature would no longer cover.

The public key is distributed out of band. You hold it, and that is what makes the check independent of the service that produced the log. Keep it with your audit records rather than fetching it at verification time: a verifier that asks the signer for the key it will verify with is not verifying much.

Two things worth building into whatever runs this:

Pin the heads you have seen. Keep each (tree_size, root_hash, signature) you verify. A later head must be consistent with an earlier one; if it is not, the log was rewritten, and the pinned pair is your evidence.
Verify on ingest, not on demand. Verifying as the entry is written is cheap and surfaces a problem while the context still exists. An audit years later is when you least want to find one.

5. Contract testing

The audit.svc OpenAPI specification is the contract. Import it into Bruno, Postman, Insomnia or any OpenAPI-compatible client, and every endpoint becomes a saved request with schemas and examples.

Bruno’s CLI runs the same collection in CI, which is the cheapest way to keep an integration honest across releases:

npm i -g @usebruno/cli
bru run path/to/audit-collection --env dev

Point it at each environment in turn. A contract test that only ever runs against dev tells you about dev.