Test Definitions

This is the schema for the Testing v2 runner (kis test / kis test run). Tests are YAML, organized in directories, with three levels:

Suite (directory or file)
  └── Test (unit of pass/fail)
        └── Step (one protocol action + assertions)

For the older steps:/testcases:/scenarios:/testplans: format used by test.svc tests, see legacy tests mode.

Discovery: directory = suite

kis test -t <path> accepts a single YAML file (one flat suite) or a directory:

• Every directory is a suite; subdirectories become child suites, recursively.
• A file named suite.yaml in a directory supplies suite-level config (name, variables, tags, data, hooks, parallel). All other *.yaml/*.yml files contribute only their tests: — file-level variables: or hooks in non-suite.yaml files are intentionally ignored.
• Files and subdirectories are processed in sorted order.
• Skipped: hidden entries (leading .), directories starting with _ (the convention for shared YAML pulled in via import:), and directories containing no YAML anywhere.
• Multiple YAML files in one directory merge into the same suite. Colliding data: names are last-write-wins.

tests/
├── suite.yaml            # root suite config: variables, tags, hooks
├── env.yaml              # env file (auto-discovered — not a test file)
├── users/
│   ├── suite.yaml        # child suite; inherits root variables/tags
│   ├── crud.yaml         # tests
│   └── permissions.yaml  # tests
└── orders/
    └── checkout.yaml     # child suite without its own suite.yaml

Composition (import:)

Files that declare a top-level import: block are expanded through yamlplus (merge: / replace: / insert: / splice: verbs with selectors), with the file’s directory as the base for relative references. Files without import: are parsed as plain YAML. Put shared step blocks in a _-prefixed directory so they aren’t loaded as suites themselves.

Suite schema

Top-level keys of suite.yaml (or of a single-file suite):

Key	Type	Default	Meaning
name	string	directory/file basename	Suite identifier.
tags	[]string	—	Accumulate down the tree: effective tags = ancestor suite tags ∪ test tags.
variables	map	—	Visible to every test in this suite and children; deeper suite overrides shallower.
data	map[name]DataSource	—	Named data sources for table: iteration — see data-driven.
parallel	bool	false	Run this suite’s tests concurrently.
max_parallel	int	0 = unbounded	Concurrency cap when parallel: true.
tests	[]Test	—	Tests declared in this file.
before_all / before_each / after_each / after_all	[]Step	—	Hooks; same shape as test steps. See below.

Test schema

Key	Type	Required	Default	Meaning
name	string	yes	—	Unique within the suite.
tags	[]string	no	—	Combined with suite-chain tags for --tags filtering.
variables	map	no	—	Merged over the suite snapshot before hooks/steps run.
table	string	no	—	Data source name; runs the steps once per row.
steps	[]Step	yes	—	Executed in order.
continue_on_error	bool	no	false	Keep running steps after a failure (reports all failures); also lets table iteration continue past a failing row.
timeout	duration	no	none	Caps the whole test.

Step schema

Each step declares exactly one protocol key.

Key	Type	Required	Default	Meaning
name	string	yes	—	Step identifier.
http / grpc / sql / script	object	exactly one	—	Protocol config (below).
assertions	[]string	no	—	Expressions evaluated after the response — see assertions and variables.
continue_on_error	bool	no	false	A failing step doesn’t abort the test.
timeout	duration	no	protocol default	Per-step override.
retries	int	no	0	Extra attempts on transport errors only — assertion failures are not retried. Total attempts = 1 + retries.
retry_delay	duration	no	200ms	Constant delay between attempts.
tags	[]string	no	—	Per-step tags (reporting granularity).

HTTP steps

tests:
  - name: create-user
    steps:
      - name: create
        http:
          url: "{{baseurl}}/users"            # required
          method: POST                        # default GET
          headers:
            Content-Type: application/json
            Authorization: "Bearer {{token}}"
          query:                              # appended to the URL; existing params preserved
            page: "1"
          body:
            type: raw                         # raw (default) | form | multipart
            payload: '{"name": "{{username}}"}'
          timeout: 10s                        # default 30s
          follow_redirects: true              # default true
          tls:
            skip_verify: true                 # self-signed certs in dev
          response:
            type: json                        # json (default) | text | binary
            variables:                        # extract into scope for later steps
              user_id: "response.id"
              first_tag: "jq: .response.tags[0]"
        assertions:
          - status_code == 201
          - response.id != null
          - response.name == "{{username}}"
          - duration_ms < 500
          - headers["content-type"] contains "application/json"

Body types:

• raw — payload string sent as-is; set Content-Type yourself.
• form — fields: map → application/x-www-form-urlencoded. Scalar values only; nested maps/arrays are rejected.
• multipart — fields: values may be a scalar, a file part { file: "fixtures/avatar.png", content_type: "image/png", filename: "avatar.png" } (content_type defaults to application/octet-stream, filename to the basename), or a text part { value: "text", content_type: "..." }.

Behavior notes:

• The body builder sets Content-Type first; your headers: override it.
• Default User-Agent: kis-test/0.2-dev unless you set one.
• A non-2xx status is not an error — assert on status_code. Only transport failures (DNS, refused connection, timeout) error the step (and are what retries retries).
• Wire timings are captured per exchange: dns_ms, connect_ms, tls_ms, ttfb_ms, transfer_ms, total_ms — persisted with --db, and ttfb_ms/total_ms are assertable.
• Cookies: each test gets one cookie jar shared by all its HTTP steps and before_each/ after_each hooks, so login → authenticated-call flows work without manual cookie plumbing.
• tls.client_cert / tls.client_key / tls.root_cas parse but are not wired yet; only skip_verify is honored.

gRPC steps

steps:
  - name: get-user
    grpc:
      address: "{{grpc_host}}:443"         # required
      service: user.v1.UserService         # required, fully qualified
      method: GetUser                      # required
      tls:                                 # omit entirely for plaintext
        skip_verify: true
      metadata:
        authorization: "Bearer {{token}}"
      payload:                             # request message; snake_case or camelCase field names
        user_id: "{{user_id}}"
      timeout: 10s                         # default 30s
      response:
        variables:
          user_name: "response.user.name"
    assertions:
      - status_code == 0                   # gRPC status code, 0 = OK
      - response.user.id == "{{user_id}}"

• Server reflection only — the target server must expose the reflection service. The proto:/proto_path: fields parse but proto-file mode is not implemented yet.
• Unary calls only; streaming is out of scope for now.
• A non-OK gRPC status is not an error: it lands in status_code / status_message for assertions.
• Payloads are marshaled via protojson; responses use UseProtoNames + EmitUnpopulated, so assert with snake_case field names and expect zero-valued fields to be present.

SQL steps

steps:
  - name: verify-row
    sql:
      driver: postgres      # required: postgres|postgresql|pg, mysql, clickhouse, duckdb, sqlite|sqlite3
      connection: "postgres://user:pass@localhost:5432/db?sslmode=disable"
      query: "SELECT id, name FROM users WHERE id = $1"
      params: ["{{user_id}}"]              # positional only
      timeout: 5s                          # default 10s
      response:
        variables:
          db_name: "rows[0].name"
          total: "row_count"
    assertions:
      - row_count == 1
      - rows[0].name == "{{username}}"

  - name: seed
    sql:
      driver: postgres
      connection: "{{db_url}}"
      execute: "INSERT INTO users (id, name) VALUES ($1, $2)"
      params: ["{{user_id}}", "{{username}}"]
    assertions:
      - rows_affected == 1

• query (SELECT → rows, row_count, columns) and execute (DML → rows_affected) are mutually exclusive.
• Empty connection opens an in-memory DuckDB — handy for pure-SQL smoke tests.
• Named parameters are not supported yet; use positional params.
• rows keeps at most a 1000-row sample; []byte columns coerce to string, timestamps to RFC3339Nano.
• DB errors mark the step errored (with the error in evidence) rather than aborting the run.

Script steps

steps:
  - name: custom-check
    script:
      language: js          # default js. Also: js:v8, lua, starlark, cel, expr
      execute: |
        const r = response;                 // parsed body of the previous HTTP step
        if (!r.items.every(i => i.price > 0)) {
          test.fail("non-positive price found", 400);
        }
        test.set_variable("item_count", r.items.length);
        test.success("all items valid", 200);
      timeout: 10s          # default 5s
      params: { max: 10 }   # extra vars merged over the scope

execute (inline) and file: (path relative to the test YAML’s directory) are mutually exclusive. See scripting for the full test.* API, injected variables, and pass/fail semantics.

Hooks

Hooks are steps that run at suite boundaries; they use the identical step schema.

Hook	Runs	Failure behavior
before_all	once per suite, before its tests	suite’s tests are skipped; after_all still runs
before_each	full suite chain root → leaf, before every test (and every table row)	test marked errored, steps skipped; after_each still runs
after_each	leaf → root, after every test, pass or fail	logged, does not change the test result
after_all	once per suite, always — even after errors	logged, does not change the suite result

Variables set by before_all persist in the suite scope (visible to all tests in the suite and to child suites). Variables set by before_each live in the test scope. This is how auth tokens propagate:

suite.yaml

before_each:
  - name: authenticate
    http:
      url: "{{baseurl}}/auth"
      method: POST
      body:
        payload: '{"client_id": "{{client_id}}"}'
      response:
        variables:
          token: "response.access_token"

Suite-level hooks (before_all/after_all) get their own cookie jar; per-test hooks share the test’s jar.

Tags and filtering

kis test -t tests/ --tags smoke                 # tagged smoke
kis test -t tests/ --tags "smoke,api"           # smoke AND api
kis test -t tests/ --tags smoke --tags nightly  # smoke OR nightly
kis test -t tests/ --tags api --exclude-tags wip

Commas inside one --tags value = AND; repeating the flag = OR; --exclude-tags uses the same syntax and wins ties. A test’s effective tags are the union of its own tags and every ancestor suite’s tags.

Validation

--dry-run loads the suite tree, validates it, and resolves templates without making any HTTP/gRPC/SQL calls. Load-time validation enforces: every test has name, every step has name and exactly one protocol key; hooks are validated the same way.