AI Task Reference

The AI tasks available to flows and harnesses, with every parameter — type, whether it is required, its default — and worked examples.

Model calls resolve through the AI Gateway, so a task’s model traffic is subject to the same aliases, budgets and cost accounting as any other call on the platform.

Note

The sofia task and namespace write to AI Knowledge. The identifier predates the block’s external name; it is the string you type today, so it is documented as-is.

Chat

LLM chat and completion requests with model selection, context management, and parallel execution support.

Parameters

Parameter	Type	Required	Default	Description
type	string	No	completion	Request type: chat, completion, or gateway
prompt	string/list	Yes*	—	Prompt text(s) for completion mode. Supports Liquid templates
messages	list	Yes*	—	Chat messages for chat mode. Each entry has role and content
selection	map	No	{}	Model selection criteria: cost, time, accuracy, model
temperature	float	No	1.0	Sampling temperature
maxtokens	int	No	200	Maximum tokens to generate
choices	int	No	1	Number of response choices
setkey	string	No	—	Environment key to store the response
contextlimit	int	No	2000	Maximum context token limit
contextkey	string	No	—	Environment key for context maps
contextfield	string	No	—	Field within context maps containing context text
linksfield	string	No	—	Field within context maps containing source URLs
promptkey	string	No	—	Key to rank context maps against
stop	string	No	—	Stop token sequences
url	string	No	—	Custom endpoint URL
responsekey	string	No	—	Key to extract from response
timeout	int	No	500	Request timeout in seconds
parallel	int	No	-1	Number of parallel requests (-1 = sequential)

* prompt is required for completion mode; messages is required for chat mode.

Examples

Chat mode with messages

tasks:
  - name: ask
    chat:
      type: chat
      messages:
        - role: system
          content: "You are a helpful assistant."
        - role: user
          content: "{{user.prompt}}"
      selection:
        model: gpt-4
      maxtokens: 500
      setkey: chatresult
    outputs:
      - next

Completion mode with prompt

tasks:
  - name: complete
    chat:
      type: completion
      prompt: "Summarize the following text: {{context}}"
      selection:
        accuracy: high
      temperature: 0.7
      maxtokens: 300
      setkey: summary
    outputs:
      - next

Chat with context from vector search

tasks:
  - name: answer
    chat:
      type: chat
      messages:
        - role: system
          content: "Answer based on the provided context."
        - role: user
          content: "{{user.question}}"
      selection:
        model: gpt-4
      contextkey: searchresults
      contextfield: scraped_data
      linksfield: url
      contextlimit: 4000
      maxtokens: 1000
      setkey: answer
    outputs:
      - respond

Namespace: llm

LLM chat and completion functions.

Scope: System

Note: llm.chat and llm.complete require a pipeline environment with ProxySelector for model routing. Use the YAML interface for full functionality.

Functions

Function	Description
llm.chat()	LLM chat completion
llm.complete()	LLM text completion

Embed

Tokenize text data and generate vector embeddings in a two-step process. First tokenizes (splits) input data using a configurable tokenizer, then generates embeddings for each chunk via an embedding model endpoint.

Parameters

Parameter	Type	Required	Default	Description
data	string	Yes	—	Environment key referencing the data to embed. Supports Liquid templates
setkey	string	No	—	Environment key to store the embeddings result
type	string	No	—	Embedding type (e.g. openai) to determine API endpoint
defaulturl	string	No	data	Default key name for single-string input
tokenizer	map	Yes	—	Tokenizer configuration (see below)
embedder	map	Yes	—	Embedder configuration (see below)
timeout	int	No	500	Request timeout in seconds per embedding call

Tokenizer Configuration

Field	Type	Description
selection.model	string	Tokenizer model: string-splitter or length-splitter
params.string	string	Delimiter for string-splitter
params.length	int	Chunk size for length-splitter

Embedder Configuration

Field	Type	Description
selection	map	Model selection criteria: cost, time, accuracy, model

Examples

Tokenize and embed with string splitter

tasks:
  - name: embed-text
    embed:
      data: scraped
      setkey: embeddings
      type: openai
      tokenizer:
        selection:
          model: string-splitter
        params:
          string: "\n\n"
      embedder:
        selection:
          model: text-embedding-ada-002
    outputs:
      - store

Tokenize and embed with length splitter

tasks:
  - name: embed-text
    embed:
      data: scraped
      setkey: embeddings
      tokenizer:
        selection:
          model: length-splitter
        params:
          length: 500
      embedder:
        selection:
          model: text-embedding-ada-002
    outputs:
      - store

Namespace: embed

See the embed namespace documentation and the embeddata task for related functionality.

EmbedData

Generate vector embeddings from text data via a REST embedding API. Supports splitting input data by delimiter or character length before embedding.

Parameters

Parameter	Type	Required	Default	Description
data	string	Yes	—	Environment key referencing the data to embed. Supports Liquid templates
setkey	string	No	—	Environment key to store the embeddings result
selection	map	No	{}	Model selection criteria: cost, time, accuracy, model
type	string	No	—	Embedding type (e.g. openai) to determine API endpoint
splitter	string	No	—	Delimiter string to split data before embedding
limit	int	No	0	Character length to split data into chunks
defaulturl	string	No	data	Default key name for single-string input
timeout	int	No	500	Request timeout in seconds

Examples

Generate embeddings with string splitter

tasks:
  - name: generate-embeddings
    embeddata:
      data: scraped
      selection:
        model: text-embedding-ada-002
      type: openai
      splitter: "\n\n"
      setkey: embeddings
    outputs:
      - store

Generate embeddings with length splitter

tasks:
  - name: generate-embeddings
    embeddata:
      data: scraped
      selection:
        model: text-embedding-ada-002
      limit: 1000
      setkey: embeddings
    outputs:
      - store

Namespace: embed

Embedding generation and text splitting functions.

Scope: System

Note: embed.generate requires a pipeline environment with ProxySelector. The embed.split function works without a pipeline environment.

Functions

Function	Description
embed.generate()	Generate embeddings (requires pipeline)
embed.split()	Split text data by delimiter or length

Guard (Pre-Guard / Post-Guard)

LLM safety classification using Llama Guard. Classifies user prompts (pre-guard) or agent responses (post-guard) against unsafe content categories (S1–S11). Returns safe, unsafe (with violated categories), or unknown.

Parameters

Parameter	Type	Required	Default	Description
user-prompt	string	Yes*	—	User prompt to classify (pre-guard). Supports Liquid templates
agent-response	string	Yes*	—	Agent response to classify (post-guard). Supports Liquid templates
selection	map	No	{}	Model selection criteria for the guard model
unsafe-categories	list	No	all (S1–S11)	Subset of categories to check
error-response	string	No	—	Response message when content is unsafe. Supports Liquid templates
error-trigger	string	No	—	JavaScript code to execute when content is unsafe
setkey	string	No	—	Environment key to store the guard result
continueonerror	bool	No	true	If true, returns success even when content is unsafe
temperature	float	No	1.0	Guard model temperature
maxtokens	int	No	200	Guard model max tokens
contextlimit	int	No	2000	Maximum context token limit
timeout	int	No	500	Request timeout in seconds

* At least one of user-prompt or agent-response is required.

Unsafe Content Categories

Category	Description
S1	Violent Crimes
S2	Non-Violent Crimes
S3	Sex Crimes
S4	Child Exploitation
S5	Specialized Advice
S6	Privacy
S7	Intellectual Property
S8	Indiscriminate Weapons
S9	Hate
S10	Self-Harm
S11	Sexual Content

Task Keys

The guard task is registered under two task keys:

Task Key	Usage
pre-guard	Check user input before processing
post-guard	Check agent output before delivery

Examples

Pre-guard (check user input)

tasks:
  - name: safety-check
    pre-guard:
      user-prompt: "{{user.message}}"
      selection:
        model: llama-guard
      unsafe-categories:
        - S1
        - S4
        - S9
      error-response: "Your message was flagged as unsafe."
      setkey: guardresult
    outputs:
      - process

Post-guard (check agent output)

tasks:
  - name: output-check
    post-guard:
      agent-response: "{{chatresult.choices[0].message.content}}"
      selection:
        model: llama-guard
      error-response: "The response was flagged."
      setkey: guardresult
    outputs:
      - respond

Guard with error trigger

tasks:
  - name: safety-check
    pre-guard:
      user-prompt: "{{user.message}}"
      selection:
        model: llama-guard
      error-trigger: |
        log("Unsafe content detected: " + result.categories);
        setvar("blocked", "true");
      setkey: guardresult
    outputs:
      - process

Namespace: guard

LLM safety classification functions.

Scope: System

Note: guard.check requires a pipeline environment with ProxySelector for model routing. Use the YAML interface for full functionality.

Functions

Function	Description
guard.check()	Classify content safety

GetIntent

Intent classification for user prompts. Sends prompts to an intent classification model endpoint and returns classification results.

Parameters

Parameter	Type	Required	Default	Description
prompt	string/list	Yes	—	Prompt(s) to classify. Supports Liquid templates
selection	map	No	{}	Model selection criteria: cost, time, accuracy, model
timeout	int	No	500	Request timeout in seconds

Examples

Classify user intent

tasks:
  - name: classify
    getintent:
      prompt: "{{user.message}}"
      selection:
        model: intent-classifier
    outputs:
      - route

Classify with multiple prompts

tasks:
  - name: classify-batch
    getintent:
      prompt:
        - "{{user.message}}"
        - "{{user.context}}"
      selection:
        accuracy: high
    outputs:
      - route

Namespace: intent

Intent classification functions.

Scope: System

Note: intent.classify requires a pipeline environment with ProxySelector for model routing. Use the YAML interface for full functionality.

Functions

Function	Description
intent.classify()	Classify the intent of a prompt

MLPipelineChat

Stream pipeline ML chat task. Reads incoming data records from pipeline inputs, builds chat messages from conversation memory and the current record, delegates to the chat task in chat mode, and writes the parsed ML response to pipeline outputs. Designed for streaming multi-turn conversations within a data pipeline.

Registered as both mlpipelinechat and mlpipechat.

Parameters

Parameter	Type	Required	Default	Description
selection	map	No	{}	Model selection criteria: cost, time, accuracy, model
promptkey	string	No	—	Record field name containing the user prompt

Examples

Pipeline chat with model selection

tasks:
  - name: request
    mlpiperequest:
    inputs:
      - source
    outputs:
      - chat

  - name: chat
    mlpipechat:
      selection:
        model: gpt-4
      promptkey: message
    inputs:
      - request
    outputs:
      - response

Full pipeline with request, chat, and response

tasks:
  - name: ingest
    mlpiperequest:
    inputs:
      - source
    outputs:
      - process

  - name: process
    mlpipelinechat:
      selection:
        model: gpt-4
        accuracy: high
      promptkey: user_message
    inputs:
      - ingest
    outputs:
      - deliver

  - name: deliver
    mlpiperesponse: "https://api.example.com/v1/conversation/response"
    inputs:
      - process

Namespace: pipeline

See the pipeline namespace documentation for the pipeline.chat() function details.

MLPipelineRequest

Stream pipeline ML request task. Reads incoming data records from pipeline inputs and forwards them to pipeline outputs with conversation context. Acts as the request ingestion stage of a streaming ML pipeline.

Registered as both mlpipelinerequest and mlpiperequest.

Parameters

This task reads its configuration from pipeline inputs/outputs and the incoming record. No additional execute parameters are required beyond the pipeline wiring.

Examples

Pipeline request ingestion

tasks:
  - name: ingest
    mlpiperequest:
    inputs:
      - source
    outputs:
      - chat

Full pipeline with request stage

tasks:
  - name: request
    mlpiperequest:
    inputs:
      - source
    outputs:
      - process

  - name: process
    mlpipechat:
      selection:
        model: gpt-4
      promptkey: message
    inputs:
      - request
    outputs:
      - response

  - name: response
    mlpiperesponse: "https://api.example.com/v1/response"
    inputs:
      - process

Namespace: pipeline

See the pipeline namespace documentation for the pipeline.request() function details.

MLPipelineResponse

Stream pipeline ML response task. Reads processed data records from pipeline inputs and submits them to an external URL via HTTP POST with authorization headers. Acts as the response delivery stage of a streaming ML pipeline.

Registered as both mlpipelineresponse and mlpiperesponse.

Parameters

Parameter	Type	Required	Default	Description
(direct value)	string	Yes	—	The URL to submit the response to

The URL is provided as the direct value of the task key, not as a named parameter.

Examples

Pipeline response delivery

tasks:
  - name: respond
    mlpiperesponse: "https://api.example.com/v1/conversation/response"
    inputs:
      - chat

Full pipeline with response stage

tasks:
  - name: request
    mlpiperequest:
    inputs:
      - source
    outputs:
      - chat

  - name: chat
    mlpipechat:
      selection:
        model: gpt-4
      promptkey: message
    inputs:
      - request
    outputs:
      - respond

  - name: respond
    mlpipelineresponse: "https://api.example.com/v1/conversation/response"
    inputs:
      - chat

Namespace: pipeline

See the pipeline namespace documentation for the pipeline.response() function details.

ScrapeData

Scrape content from web URLs. Supports scraping websites with configurable crawl depth and page limits. Posts scrape requests to a scraping service resolved via ProxySelector. Scraped content is stored in the pipeline environment.

Parameters

Parameter	Type	Required	Default	Description
urls	list	Yes	—	List of URL specifications (strings or maps)
selection	map	No	{}	Service selection criteria: cost, time, accuracy, model
depth	int/string	No	1	Crawl depth for website scraping. Supports Liquid templates
setkey	string	No	—	Environment key to store the scraped data
limit	int	No	-1	Maximum number of pages to scrape (-1 = unlimited)
contextlimit	int	No	15000	Maximum character limit per scraped page
timeout	int	No	500	Request timeout in seconds

URL Specification (map form)

Field	Type	Description
url	string/list	One or more URLs to scrape
type	string	Scrape type: website
depth	string	Crawl depth for this URL
setkey	string	Environment key for this URL’s results
limit	int	Page limit for this URL
contextLimit	int	Character limit for this URL

Examples

Scrape a website

tasks:
  - name: scrape
    scrapedata:
      urls:
        - "https://docs.example.com/guide"
      selection:
        model: scraper
      depth: 2
      setkey: scraped
      contextlimit: 10000
    outputs:
      - embed

Scrape multiple URLs

tasks:
  - name: scrape
    scrapedata:
      urls:
        - url:
            - "https://example.com/page1"
            - "https://example.com/page2"
          type: website
          depth: "1"
          setkey: webdata
          contextLimit: 15000
    outputs:
      - process

Scrape with page limit

tasks:
  - name: scrape-limited
    scrapedata:
      urls:
        - "https://docs.example.com"
      depth: 3
      limit: 50
      setkey: scraped
    outputs:
      - embed

Namespace: scrape

See the scrape namespace documentation for the scrape.web() function details.

Scrape-DOCX

Scrape text content from DOCX files via a remote scraping service. Posts the file URL to the scraping service endpoint resolved via ProxySelector and stores the extracted text in the pipeline environment.

Parameters

Parameter	Type	Required	Default	Description
fileurl	string	Yes	—	URL of the DOCX file to scrape. Supports Liquid templates
setkey	string	No	—	Environment key to store the scraped content
selection	map	No	—	Service selection criteria for ProxySelector
contextlimit	int	No	15000	Maximum character limit for scraped content
timeout	int	No	500	Request timeout in seconds

Examples

Scrape a DOCX file

tasks:
  - name: read-docx
    scrape-docx:
      fileurl: "{{document.url}}"
      selection:
        model: scraper
      setkey: docxcontent
    outputs:
      - process

Scrape DOCX with context limit

tasks:
  - name: read-docx
    scrape-docx:
      fileurl: "https://storage.example.com/docs/manual.docx"
      setkey: docxdata
      contextlimit: 25000
    outputs:
      - embed

Namespace: scrape

See the scrape namespace documentation for the scrape.docx() function details.

Scrape-Excel

Scrape content from Excel files. Downloads the file and extracts sheet data using the worksheet library. Does not require ProxySelector — processes Excel files directly.

Parameters

Parameter	Type	Required	Default	Description
fileurl	string	Yes	—	URL of the Excel file to scrape. Supports Liquid templates
setkey	string	No	—	Environment key to store the scraped content

Examples

Scrape an Excel file

tasks:
  - name: read-excel
    scrape-excel:
      fileurl: "{{document.url}}"
      setkey: exceldata
    outputs:
      - process

Scrape Excel with static URL

tasks:
  - name: read-spreadsheet
    scrape-excel:
      fileurl: "https://storage.example.com/data/report.xlsx"
      setkey: reportdata
    outputs:
      - embed

Namespace: scrape

See the scrape namespace documentation for the scrape.excel() function details.

Scrape-PDF

Scrape text content from PDF files via a remote scraping service. Posts the file URL to the scraping service endpoint resolved via ProxySelector and stores the extracted text in the pipeline environment.

Parameters

Parameter	Type	Required	Default	Description
fileurl	string	Yes	—	URL of the PDF file to scrape. Supports Liquid templates
setkey	string	No	—	Environment key to store the scraped content
selection	map	No	—	Service selection criteria for ProxySelector
contextlimit	int	No	15000	Maximum character limit for scraped content
timeout	int	No	500	Request timeout in seconds

Examples

Scrape a PDF file

tasks:
  - name: read-pdf
    scrape-pdf:
      fileurl: "{{document.url}}"
      selection:
        model: scraper
      setkey: pdfcontent
      contextlimit: 20000
    outputs:
      - embed

Scrape PDF with template URL

tasks:
  - name: read-pdf
    scrape-pdf:
      fileurl: "https://storage.example.com/docs/{{document.filename}}"
      setkey: pdfdata
    outputs:
      - process

Namespace: scrape

See the scrape namespace documentation for the scrape.pdf() function details.

Scrape-PPTX

Scrape text content from PPTX (PowerPoint) files via a remote scraping service. Posts the file URL to the scraping service endpoint resolved via ProxySelector and stores the extracted text in the pipeline environment.

Parameters

Parameter	Type	Required	Default	Description
fileurl	string	Yes	—	URL of the PPTX file to scrape. Supports Liquid templates
setkey	string	No	—	Environment key to store the scraped content
selection	map	No	—	Service selection criteria for ProxySelector
contextlimit	int	No	15000	Maximum character limit for scraped content
timeout	int	No	500	Request timeout in seconds

Examples

Scrape a PPTX file

tasks:
  - name: read-pptx
    scrape-pptx:
      fileurl: "{{document.url}}"
      selection:
        model: scraper
      setkey: pptxcontent
    outputs:
      - embed

Scrape PPTX with template URL

tasks:
  - name: read-presentation
    scrape-pptx:
      fileurl: "https://storage.example.com/slides/{{presentation.id}}.pptx"
      setkey: slidecontent
      contextlimit: 20000
    outputs:
      - process

Namespace: scrape

See the scrape namespace documentation for the scrape.pptx() function details.

Sofia

Sofia vector store operations. Sends embeddings and associated data to a Sofia vector store REST endpoint. Iterates over embeddings from the pipeline environment, templates a payload for each, and POSTs to the configured URL with custom headers. Supports retry configuration.

Parameters

Parameter	Type	Required	Default	Description
url	string	Yes	—	Sofia REST endpoint URL. Supports Liquid templates
embeddingskey	string	Yes	—	Environment key containing the embeddings data (with .embeddings, .keys, .data sub-keys)
payload	string	No	—	Liquid template string for the POST body
headers	map	No	—	HTTP headers to include. Values support Liquid templates
optype	string	No	insert	Operation type. Supports Liquid templates
doretry	string/bool	No	false	Whether to enable HTTP retry

Payload Template Variables

The payload template has access to these variables:

Variable	Description
embeddings_data	The text/data for the current embedding
embeddings_key	The key/source for the current embedding
embeddings	The embedding vector (as JSON array)

Examples

Store embeddings in Sofia

tasks:
  - name: store-in-sofia
    sofia:
      url: "https://sofia.example.com/v1/documents"
      embeddingskey: embeddings
      payload: |
        {
          "text": "{{embeddings_data}}",
          "source": "{{embeddings_key}}",
          "vector": {{embeddings}}
        }
      headers:
        Authorization: "Bearer {{user.token}}"
        Content-Type: "application/json"
    outputs:
      - done

Store with retry enabled

tasks:
  - name: store-in-sofia
    sofia:
      url: "https://sofia.example.com/v1/documents"
      embeddingskey: embeddings
      optype: upsert
      doretry: true
      headers:
        Authorization: "Bearer {{api.token}}"
    outputs:
      - done

Namespace: sofia

See the sofia namespace documentation for the sofia.embed() function details.

TopicsDelete

Delete records from a Milvus collection by user ID, topic name, and optional URL filter. Uses the Milvus Delete API with expression filters.

Parameters

Parameter	Type	Required	Default	Description
url	string	Yes	—	Milvus gRPC endpoint URL
collection	string	Yes	—	Milvus collection name. Supports Liquid templates
userid	string	Yes	—	User ID to delete by. Supports Liquid templates
topic	string	Yes	—	Topic name to delete by. Supports Liquid templates
link	string	No	—	URL to filter by (delete only matching URL). Supports Liquid templates

Examples

Delete a topic

tasks:
  - name: remove-topic
    topicsdelete:
      url: "localhost:19530"
      collection: "knowledge_base"
      userid: "{{user.id}}"
      topic: "{{topic.name}}"
    outputs:
      - done

Delete a specific URL from a topic

tasks:
  - name: remove-url
    topicsdelete:
      url: "localhost:19530"
      collection: "knowledge_base"
      userid: "{{user.id}}"
      topic: "docs"
      link: "https://docs.example.com/old-page"
    outputs:
      - done

Namespace: milvus

See the milvus namespace documentation for the milvus.delete() function details.

TopicsInsert

Insert vector embeddings into a Milvus collection. Reads pre-computed embeddings, keys, and data arrays from the pipeline environment and inserts them with user ID, topic name, URL, and scraped data columns. Uses tenant-based database isolation.

Parameters

Parameter	Type	Required	Default	Description
url	string	Yes	—	Milvus gRPC endpoint URL
collection	string	Yes	—	Milvus collection name. Supports Liquid templates
embeddings	string	Yes	—	Environment key containing the embeddings data (with .embeddings, .keys, .data sub-keys)
userid	string	Yes	—	User ID for the inserted records. Supports Liquid templates
topic	string	Yes	—	Topic name for the inserted records. Supports Liquid templates
topicfield	string	No	topic_name	Milvus field name for the topic column

Examples

Insert embeddings into Milvus

tasks:
  - name: store-vectors
    topicsinsert:
      url: "localhost:19530"
      collection: "knowledge_base"
      embeddings: embeddingresult
      userid: "{{user.id}}"
      topic: "{{topic.name}}"
    outputs:
      - done

Insert with custom topic field

tasks:
  - name: store-vectors
    topicsinsert:
      url: "localhost:19530"
      collection: "documents"
      embeddings: embeddings
      userid: "{{user.id}}"
      topic: "{{category}}"
      topicfield: category_name
    outputs:
      - done

Namespace: milvus

See the milvus namespace documentation for the milvus.insert() function details.

TopicsList

List unique topic names from a Milvus collection for a given user. Loads the collection, queries by user ID with optional topic and URL filters, and returns deduplicated topic names with optional scraped data.

Parameters

Parameter	Type	Required	Default	Description
url	string	Yes	—	Milvus gRPC endpoint URL
collection	string	Yes	—	Milvus collection name. Supports Liquid templates
userid	string	Yes	—	User ID to filter by. Supports Liquid templates
topic	string	No	—	Topic name to filter by. Supports Liquid templates
link	string	No	—	URL to filter by. Supports Liquid templates
senddata	bool	No	false	Whether to include scraped data in results
setkey	string	No	—	Environment key to store the list results

Examples

List all topics for a user

tasks:
  - name: list-topics
    topicslist:
      url: "localhost:19530"
      collection: "knowledge_base"
      userid: "{{user.id}}"
      setkey: topicnames
    outputs:
      - display

List topics with data included

tasks:
  - name: list-topics-data
    topicslist:
      url: "localhost:19530"
      collection: "knowledge_base"
      userid: "{{user.id}}"
      senddata: true
      setkey: topicdetails
    outputs:
      - process

Namespace: milvus

See the milvus namespace documentation for the milvus.list() function details.

TopicsQuery

Query records from a Milvus collection using field-based filters (not vector similarity). Loads the collection and queries by user ID, topic name, and optional URL filter. Returns matching records with ID, URL, and scraped data.

Parameters

Parameter	Type	Required	Default	Description
url	string	Yes	—	Milvus gRPC endpoint URL
collection	string	Yes	—	Milvus collection name. Supports Liquid templates
topic	string	Yes	—	Topic name to query by. Supports Liquid templates
userid	string	No	—	User ID to filter by. Supports Liquid templates
topicurl	string	No	—	URL to filter by. Supports Liquid templates
limit	int	No	3	Maximum number of results
setkey	string	No	—	Environment key to store the query results

Examples

Query records by topic

tasks:
  - name: query-topics
    topicsquery:
      url: "localhost:19530"
      collection: "knowledge_base"
      topic: "{{topic.name}}"
      userid: "{{user.id}}"
      limit: 10
      setkey: queryresults
    outputs:
      - process

Query with URL filter

tasks:
  - name: query-by-url
    topicsquery:
      url: "localhost:19530"
      collection: "knowledge_base"
      topic: "docs"
      topicurl: "https://docs.example.com"
      limit: 20
      setkey: queryresults
    outputs:
      - process

Namespace: milvus

See the milvus namespace documentation for the milvus.query() function details.

TopicsSearch

Perform vector similarity search in a Milvus collection. Loads the collection, applies user/topic filters, and searches using L2 distance against a query embedding vector. Returns matched records with ID, URL, and scraped data.

Parameters

Parameter	Type	Required	Default	Description
url	string	Yes	—	Milvus gRPC endpoint URL
collection	string	Yes	—	Milvus collection name. Supports Liquid templates
embeddings	string	Yes	—	Environment key containing the query embedding vector
topic	string	Yes	—	Topic name to filter by. Supports Liquid templates
userid	string	No	—	User ID to filter by. Supports Liquid templates
topicurl	string	No	—	URL to filter by. Supports Liquid templates
limit	int	No	3	Maximum number of results (top-k)
setkey	string	No	—	Environment key to store the search results

Examples

Vector similarity search

tasks:
  - name: search-similar
    topicssearch:
      url: "localhost:19530"
      collection: "knowledge_base"
      embeddings: queryembedding
      topic: "{{topic.name}}"
      userid: "{{user.id}}"
      limit: 5
      setkey: searchresults
    outputs:
      - respond

Search without user filter

tasks:
  - name: search-global
    topicssearch:
      url: "localhost:19530"
      collection: "knowledge_base"
      embeddings: queryembedding
      topic: "general"
      limit: 10
      setkey: searchresults
    outputs:
      - process

Namespace: milvus

See the milvus namespace documentation for the milvus.search() function details.

ToSentence

Convert entity YAML definitions into natural language sentences. Parses entity YAML using the parser/transform library and produces human-readable sentence representations. Useful for turning structured entity definitions into prompts or descriptions.

Parameters

Parameter	Type	Required	Default	Description
entity	string	Yes	—	Entity YAML string to convert. Supports Liquid templates
setkey	string	No	mlresponse	Environment key to store the resulting sentence

Examples

Convert entity YAML to sentence

tasks:
  - name: describe
    tosentence:
      entity: "{{entityyaml}}"
      setkey: entitydescription
    outputs:
      - prompt

Convert with inline entity definition

tasks:
  - name: describe-entity
    tosentence:
      entity: |
        name: User
        fields:
          - name: email
            type: string
          - name: age
            type: integer
      setkey: description
    outputs:
      - next

Namespace: text

Text transformation functions.

Scope: System

Note: text.tosentence requires entity YAML context from the pipeline environment. The text.split function works independently.

Functions

Function	Description
text.tosentence()	Convert entity YAML to sentences (requires pipeline)
text.split()	Split text by delimiter or length

Validate-JSON

Validate a JSON value against a JSON schema. Schema and value can be provided inline or referenced from the pipeline environment by key. All values support Liquid template rendering.

Parameters

Parameter	Type	Required	Default	Description
jsonschema	string	Yes*	—	Inline JSON schema string
jsonschemakey	string	Yes*	—	Environment key referencing the JSON schema
jsonvalue	string	Yes*	—	Inline JSON value string to validate
jsonvaluekey	string	Yes*	—	Environment key referencing the JSON value

* One of jsonschema or jsonschemakey is required. One of jsonvalue or jsonvaluekey is required.

Examples

Validate with inline schema and value

tasks:
  - name: check-schema
    validate-json:
      jsonschema: |
        {
          "type": "object",
          "properties": {
            "name": {"type": "string"},
            "age": {"type": "integer"}
          },
          "required": ["name"]
        }
      jsonvalue: "{{user.input}}"
    outputs:
      - process

Validate using environment keys

tasks:
  - name: check-schema
    validate-json:
      jsonschemakey: schema_definition
      jsonvaluekey: parsed_input
    outputs:
      - process

Namespace: validate

See the validate namespace documentation for the validate.json() function details.

Validate-YAML

Validate a YAML value against a JSON schema. The YAML value is parsed and validated using the JSON Schema specification. Schema and value can be provided inline or referenced from the pipeline environment by key. All values support Liquid template rendering.

Parameters

Parameter	Type	Required	Default	Description
yamlschema	string	Yes*	—	Inline YAML schema string (JSON Schema format in YAML)
yamlschemakey	string	Yes*	—	Environment key referencing the YAML schema
yamlvalue	string	Yes*	—	Inline YAML value string to validate
yamlvaluekey	string	Yes*	—	Environment key referencing the YAML value

* One of yamlschema or yamlschemakey is required. One of yamlvalue or yamlvaluekey is required.

Examples

Validate with inline schema and value

tasks:
  - name: check-yaml
    validate-yaml:
      yamlschema: |
        type: object
        properties:
          name:
            type: string
          version:
            type: string
        required:
          - name
      yamlvalue: "{{config.yaml}}"
    outputs:
      - process

Validate using environment keys

tasks:
  - name: check-yaml
    validate-yaml:
      yamlschemakey: config_schema
      yamlvaluekey: parsed_config
    outputs:
      - process

Namespace: validate

See the validate namespace documentation for the validate.yaml() function details.