Assertions & Variables

How the v2 runner (kis test run) evaluates assertions, resolves {{variables}}, and passes data between steps. Suites in the hierarchy format run with kis test tests; see the CLI Reference.

Assertions

Each entry under a step’s assertions: is one expression:

assertions:
  - status_code == 200
  - response.id != null
  - duration_ms < 500
  - response.items.length > 0
  - response.email contains "@kis.ai"
  - response.type starts_with "user."
  - headers["content-type"] contains "application/json"
  - response.name == "{{username}}"
  - "jq: [.response.users[].id] | length > 2"

All assertions in a step are evaluated (no short-circuit); any failure fails the step. Failures report expected vs. actual per expression.

Comparing numbers: leave the template unquoted

== compares types as well as values, so a number and a string are never equal. Templates are rendered before the expression is parsed, which makes the quotes decide what you are comparing:

- rows[0].id == {{user_id}}       # renders to `rows[0].id == 42`  → number vs number ✓
- rows[0].id == "{{user_id}}"     # renders to `rows[0].id == "42"` → number vs string ✗

Quote the template when the value is a string (response.tier == "{{expected}}"), leave it bare when it is a number. Get it wrong and the failure names both types rather than printing two values that look identical:

expected 42849 (string), got 42849 (int64) — same text, different types; drop the quotes to compare as a number

Grammar

<lhs-path> <operator> [<rhs>]        # comparison form
jq: <jq expression>                  # free-form jq; truthy output passes

Binary operators: ==, !=, <, <=, >, >=, contains, not_contains, starts_with, ends_with, matches (Go regexp), matches_file (tabular file comparison, below), includes (array alias of contains), has_key, not_has_key.

Unary operators (no RHS): is_string, is_number, is_boolean (alias is_bool), is_object, is_array, is_null.

LHS paths: dotted segments (response.user.id), bracketed string keys (headers["content-type"]), array indexes in either form (rows[0].name or rows.0.name), and a .length pseudo-segment on arrays and strings. Map-key lookup falls back to case-insensitive matching (useful for headers).

RHS: a double-quoted string (\" and \\ escapes), a number (123, 1.5, -3.2e2), true/false/null (case-insensitive), or anything else is treated as a path reference, so response.a == response.b and response.id == user_id (scope variable) both work.

Semantics: == is JSON-style loose equality: numeric types compare across int/float; slices/maps compare by JSON stringification. </> comparisons require numerics. contains means substring on strings, element membership on arrays, key presence on objects. jq truthiness: nil, false, "", 0, empty array/map are falsy.

What’s visible to assertions (context root)

Always: duration_ms and every scope variable (non-colliding names at top level). Per protocol:

Protocol	Fields
HTTP	response (JSON-parsed body; raw string fallback), status_code, headers (lower-cased keys), response_size, request_size, ttfb_ms, total_ms
SQL	rows, row_count, rows_affected, columns (all usable as arrays: columns contains "id", columns.length, columns[0])
gRPC	response, status_code, status_message, metadata, total_ms
CLI	exit_code, stdout, stderr, output, stdout_lines, stderr_lines, json (parsed stdout; absent when not JSON), duration_ms, cpu_ms, max_rss_mb, signal, timed_out, truncated, command
script	scope variables + duration_ms only
ws / sse	message_count, messages, connected, status_code (sse), plus every name a save_as: captured
any step with files:	files.<name>.row_count, .columns, .rows[n].<col>, .column.<col>.{sum,avg,min,max,count,empty,distinct,numeric,values}

A stream’s expect: entries carry their own expressions, evaluated against one message rather than the step: match: selects which message is meant, and assertions: validate it. Both are rooted at message (or event, the same thing): fields of a JSON payload sit at the top level, and the envelope is always available as _seq, _offset_ms, _kind, _id and _raw. A message failing match is skipped as unrelated traffic; one that matched but fails its assertions fails the step. See Test definitions.

The left side is a path, never a literal

<lhs> is a field or variable path; the literal goes on the right. A quoted left side is a load-time error rather than something that quietly resolves to nothing:

- '"{{email}}" == "[email protected]"'   # rejected: drop the quotes
- 'email == "[email protected]"'         # this

Templates inside assertions

Assertion strings are Liquid-rendered against the scope before parsing, so response.email == "{{expected_email}}" compares against the resolved value.

A rendered string value is escaped for the literal it lands in, so a value containing a quote or a backslash (O'Brien said "hi") still compares as one string, in both quote styles and in jq: expressions. A template naming a variable the scope does not hold fails the assertion with undefined variable "<name>".

js: and cel: prefixes are not supported as assertions and fail with a message saying so. Use jq: for complex expressions, or a script step for full logic.

matches_file: comparing tabular files

matches_file compares a file a step produced against a fixture, under the comparison rules on the step’s files: declaration (key:, ignore:, tolerance:):

files:
  report:
    path: "out/report.csv"
    key: ["tenant_id"]
    ignore: ["generated_at"]
assertions:
  - files.report matches_file "fixtures/expected-report.csv"

It reports a diff naming the rows and cells that differ, not a boolean. CSV, TSV and XLSX all load, and the two sides need not be the same format. See Test definitions for the full path list (files.X.row_count, files.X.column.<col>.sum, …).

Note matches and matches_file are different operators: matches is a regex over a string, matches_file is a comparison between two tables.

Templating

Interpolation uses Liquid syntax: {{ var }}, {{ var | filter }}, {% ... %} tags. Standard Liquid filters (upcase, downcase, default, json, split, …) are available.

Rendering happens at execute time, not load time: {{token}} sees variables extracted by earlier steps in the same test.

A template that names a variable the scope does not hold is an error, not an empty string:

template: url: liquid render "/users/{{usre_id}}": undefined variable "usre_id"
(did you mean "user_id"?); use {{ usre_id | default: "" }} if it is optional

Mark an optional variable with the default filter: {{ token | default: "" }}. A variable that is set to null is defined and renders as an empty string. Templates that use {% %} tags are rendered by Liquid as written.

What gets rendered, per protocol:

• HTTP: method, url, header values, query values, body.payload. Map keys are never templated. A rendered method is what lets a data-driven table carry the verb per row (method: "{{verb}}").
• SQL: connection, query, execute, and string elements of params.
• gRPC: address, service, method, metadata values, and every string inside payload recursively (nested maps/arrays included).
• Assertions: the whole expression string (see above).

variables: blocks are rendered until they stop changing, so keys in one block can reference each other in any order and through chains:

variables:
  scratch_ns: "run-{{run_id}}"
  scratch_path: "customer/{{scratch_ns}}/data"   # works, order-independent

A value that does not render (it names an undefined variable, or refers to itself) is reported on stderr and kept as written; the step that uses it then reports the template error.

Variable scope

Precedence, low → high:

1. Suite chain variables:: root first, deeper suite overrides shallower.
2. Env file (--env or auto-discovered) merged into the root suite.
3. --vars key=value: highest static precedence.
4. before_all mutations: persist in the suite scope; visible to all tests in that suite and to child suites.
5. Shared memory (export: / test.set_suite_var / test.set_run_var): run store first, then suite store. A deliberate publish outranks a static declaration of the same name.
6. Test variables:: merged over a snapshot of the suite scope. Test mutations never leak back to the suite.
7. Table row fields: override test variables per iteration (Data-driven tests).
8. before_each mutations, then step extractions as the test runs.

Shared memory: handing data to a later test

A test’s own scope dies with the test, by design. Two stores outlive it, and a step writes to them explicitly:

Store	Lifetime	Visible to
suite	one suite	later tests in that suite
run	the whole run	every suite in the run

Sibling suites do not see each other’s suite store: cross-suite data goes in the run store, deliberately.

Declaratively, from a response

Add export: beside variables: in any protocol’s response: block. The values still land in the test’s own scope; export additionally publishes them:

steps:
  - name: list-users
    http:
      url: "{{baseurl}}/users"
      response:
        export: suite                 # suite | run
        variables:
          user_rows: "jq: .response.users"

An export: naming anything else fails at load, because the symptom of a typo, a later test not seeing the data, points nowhere near the cause.

From a script

test.set_suite_var("threshold", 42);
test.set_run_var("tenant", "acme");

test.get_suite_var("threshold");
test.get_run_var("tenant");

test.set_variable still writes to the test’s own scope and is gone when the test ends; these publish deliberately to a wider one.

Feeding a dynamic table

Published rows are what a later test’s table: reads, which is the main reason to publish:

tests:
  - name: fetch-users
    steps:
      - name: list
        http:
          url: "{{baseurl}}/users"
          response:
            export: suite
            variables:
              user_rows: "jq: .response.users"

  - name: per-user                    # runs after fetch-users
    table: "{{user_rows}}"            # sees what it published

Under parallel: true

Reads and writes are safe: the stores are mutex-guarded and the suite runs clean under -race. What is not defined is ordering: two tests writing the same key race, and whether a third sees either depends on scheduling. Tables are resolved before the parallel tests dispatch, so a table: in a parallel suite reads what the suite’s hooks and prior suites published, not what a sibling test publishes.

For something every test must see, publish from before_all: that finishes before any test starts, in both engines, parallel or not.

Extracting variables from responses

Declare under the protocol’s response.variables:: extracted values merge into the test scope for subsequent steps and assertions:

http:
  url: "{{baseurl}}/auth"
  method: POST
  response:
    variables:
      token: "response.access_token"        # dotted path
      first_id: "jq: .response.users[0].id" # jq expression

Protocol	Path forms
HTTP	dotted path over response / status_code / headers, or jq:. Dotted paths cannot index arrays: use jq: for that.
CLI	dotted path over exit_code / stdout / stderr / json / duration_ms, or jq:, regex: (capture group 1), line: n (negative counts from the end).
SQL	dotted + indexed paths over rows / row_count / rows_affected / columns: rows[0].name works.
gRPC	dotted or jq: over response / status_code / status_message / metadata / duration_ms.
script	test.set_variable(key, value) writes directly to the test scope.

Extraction errors are warnings, not failures: assert on the field too if its presence matters.

When the body is not JSON at all, an HTML error page, a plain-text 500, a Go error printed verbatim, there is nothing for jq: to select from, and the warning prints the body:

↳ extract warning: extract errors: asset_id: the response body is not JSON, so `jq:` has
  nothing to select from.
      status 500, body: "storage unavailable: failed to store asset (retry after 30s)"

That body is usually the whole diagnosis, so the warning prints it in full.

Chaining example

tests:
  - name: user-lifecycle
    steps:
      - name: login
        http:
          url: "{{baseurl}}/auth/login"
          method: POST
          body:
            payload: '{"email": "{{email}}", "password": "{{password}}"}'
          response:
            variables:
              token: "response.access_token"
        assertions:
          - status_code == 200
          - response.access_token != null

      - name: create
        http:
          url: "{{baseurl}}/users"
          method: POST
          headers:
            Authorization: "Bearer {{token}}"
          body:
            payload: '{"name": "{{username}}"}'
          response:
            variables:
              user_id: "response.id"
        assertions:
          - status_code == 201

      - name: verify
        http:
          url: "{{baseurl}}/users/{{user_id}}"
          method: GET
          headers:
            Authorization: "Bearer {{token}}"
        assertions:
          - status_code == 200
          - response.id == user_id            # RHS path reference to a scope variable
          - response.name == "{{username}}"

Cookie-based auth needs no explicit chaining at all: each test has one cookie jar shared across its HTTP steps, so a login step’s Set-Cookie is replayed automatically.