Files & Transfer

Filesystem operations, archives, and moving bytes between hosts and object stores. Every parameter with its type, default and description, plus worked examples.

File

Read and write files.

Parameters

Parameter	Type	Required	Default	Description
operation	string	Yes	-	Operation: read or write
file	string	Yes	-	File path
content	string	No	-	Content to write (for write operation)
template	string	No	-	Liquid template to render and write
append	bool	No	false	Append to file instead of overwrite
setvar	string	No	-	Variable to store file content (for read)

Examples

Read a File

name: read-file
tasks:
  - name: load-config
    file:
      operation: read
      file: /config/settings.json
      setvar: settings

Write Content

name: write-file
tasks:
  - name: save-result
    file:
      operation: write
      file: /output/result.txt
      content: "Hello, World!"

Write with Template

name: write-template
vars:
  host: localhost
  port: 8080
tasks:
  - name: generate-config
    file:
      operation: write
      file: /output/config.yaml
      template: |
        server:
          host: {{host}}
          port: {{port}}

Namespace: file

File system operations.

Functions

Function	Description
file.read()	Read file contents
file.write()	Write content to file
file.exists()	Check if file/directory exists
file.remove()	Delete file or directory
file.copy()	Copy file
file.move()	Move/rename file
file.mkdir()	Create directory

Read Parameters

Parameter	Type	Default	Description
path	string	-	File path to read (required)

Write Parameters

Parameter	Type	Default	Description
path	string	-	File path to write (required)
content	string	-	Content to write (required)
append	bool	false	Append instead of overwrite
create	bool	true	Create file if not exists

Copy/Move Parameters

Parameter	Type	Default	Description
source	string	-	Source file path (required)
destination	string	-	Destination file path (required)
overwrite	bool	false	Overwrite if exists

Examples

// Read file content
let result = file.read({ path: "/config/settings.json" })
if (result.success) {
    log("Content:", result.content)
}

// Write file
file.write({
    path: "/output/result.txt",
    content: "Hello, World!",
    create: true
})

// Append to file
file.write({
    path: "/logs/app.log",
    content: new Date() + " - Log entry\n",
    append: true
})

// Check if file exists
let result = file.exists({ path: "/config/app.yaml" })
if (result.exists) {
    log("File exists, isDir:", result.isDir)
}

// Copy file
file.copy({
    source: "/source/file.txt",
    destination: "/backup/file.txt",
    overwrite: true
})

// Move/rename file
file.move({
    source: "/old/path.txt",
    destination: "/new/path.txt"
})

// Create directory
file.mkdir({
    path: "/data/output/reports",
    recursive: true
})

// Remove file or directory
file.remove({
    path: "/temp/cache",
    recursive: true
})

Glob

Find files matching glob patterns.

Parameters

Parameter	Type	Required	Default	Description
pattern	string	Yes	-	Glob pattern (e.g., **/*.go)
path	string	No	.	Base directory to search
exclude	list	No	-	Patterns to exclude
filesOnly	bool	No	false	Return only files
dirsOnly	bool	No	false	Return only directories
setvar	string	Yes	-	Variable to store matches

Examples

Find All Go Files

name: find-go-files
tasks:
  - name: search
    glob:
      pattern: "**/*.go"
      path: /project/src
      setvar: go_files

Find with Exclusions

name: find-js-files
tasks:
  - name: search
    glob:
      pattern: "**/*.js"
      path: /app
      exclude:
        - "**/node_modules/**"
        - "**/*.test.js"
      setvar: source_files

Namespace: glob

File pattern matching functions.

Functions

Function	Description
glob.find()	Find files matching glob pattern

Parameters

Parameter	Type	Default	Description
pattern	string	-	Glob pattern (e.g., **/*.go)
path	string	"."	Base directory to search
exclude	array	-	Patterns to exclude
filesOnly	bool	false	Return only files
dirsOnly	bool	false	Return only directories

Examples

// Find all Go files
let files = glob.find({
  pattern: "**/*.go",
  path: "/app",
  exclude: ["**/vendor/**"]
})
log("Found", files.matches.length, "Go files")

// Find JavaScript files, excluding tests
let jsFiles = glob.find({
  pattern: "**/*.js",
  path: "/app/src",
  exclude: ["**/*.test.js", "**/node_modules/**"],
  filesOnly: true
})

// Find all directories
let dirs = glob.find({
  pattern: "**/",
  path: "/app",
  dirsOnly: true
})

Archive

Compress and extract ZIP, TAR, TAR.GZ, and GZ archives.

Parameters

Parameter	Type	Required	Default	Description
operation	string	Yes	-	Operation: extract or compress
source	string	Yes*	-	Source file/directory
sources	list	No	-	Multiple source paths (for compress)
destination	string	Yes	-	Destination path
format	string	No	auto-detect	Format: zip, tar, tar.gz, gz
stripRoot	bool	No	false	Strip root directory when extracting
setvar	string	No	-	Variable to store output path

Examples

Extract ZIP

name: extract-zip
tasks:
  - name: unpack
    archive:
      operation: extract
      source: /downloads/package.zip
      destination: /app
      setvar: extracted_path

Extract with Strip Root

name: extract-strip
tasks:
  - name: extract-node
    archive:
      operation: extract
      source: /downloads/node-v20.0.0-linux-x64.tar.gz
      destination: /opt/node
      stripRoot: true

Create tar.gz with Multiple Sources

name: create-bundle
tasks:
  - name: bundle
    archive:
      operation: compress
      format: tar.gz
      sources:
        - /app/bin
        - /app/config
        - /app/scripts
      destination: /output/app-bundle.tar.gz

Namespace: archive

Archive extraction and compression functions.

Functions

Function	Description
archive.extract()	Extract archive file
archive.compress()	Create archive from files

Extract Parameters

Parameter	Type	Default	Description
source	string	-	Path to archive file
destination	string	-	Extraction destination
format	string	auto-detect	Archive format
stripRoot	bool	false	Strip root directory

Compress Parameters

Parameter	Type	Default	Description
source	string	-	Single source path
sources	array	-	Multiple source paths
destination	string	-	Output archive path
format	string	auto-detect	Archive format

Examples

// Extract archive
let result = archive.extract({
  source: "/downloads/package.tar.gz",
  destination: "/app",
  stripRoot: true
})
log("Extracted to:", result.path)

// Create tar.gz archive
archive.compress({
  sources: ["/app/bin", "/app/config"],
  destination: "/output/app-bundle.tar.gz",
  format: "tar.gz"
})

// Create zip from single directory
archive.compress({
  source: "/app/dist",
  destination: "/releases/app-v1.0.0.zip"
})

Extract

Extract structured content from documents (Markdown, HTML, etc.) into workflow variables.

Parameters

Parameter	Type	Required	Default	Description
format	string	No	markdown	Input format: markdown
content	string	No*	-	Inline content to extract from
file	string	No*	-	File path to read content from
glob	string	No*	-	Glob pattern for multiple files
select	string	No	-	CSS/path selector to narrow extraction
language	string	No	-	Filter code blocks by language
heading	string	No	-	Filter sections by heading text
index	int	No	-	Select a specific match by index
setvar	string	No	-	Variable to store extracted content
extractions	map	No	-	Multiple extractions (variable name to extraction spec)

*At least one of content, file, or glob is required.

Examples

Extract from Inline Content

name: extract-inline
tasks:
  - name: extract-code
    extract:
      content: |
        # README
        ```yaml
        key: value
        ```
      language: yaml
      setvar: extracted_yaml

Extract from File

name: extract-file
tasks:
  - name: extract-sections
    extract:
      file: ./docs/README.md
      heading: "Installation"
      setvar: install_instructions

Multiple Extractions

name: extract-multi
tasks:
  - name: extract-all
    extract:
      file: ./docs/guide.md
      extractions:
        install_steps:
          heading: "Installation"
        config_example:
          language: yaml
          index: 0
        api_reference:
          heading: "API"

Namespace: extract

Content extraction from structured documents.

Functions

Function	Description
extract.markdown()	Extract content from Markdown documents

Examples

// Extract code blocks by language
let result = extract.markdown({
  content: readmeContent,
  language: "yaml"
})
log("Found", result.blocks.length, "YAML blocks")

// Extract section by heading
let section = extract.markdown({
  file: "./docs/README.md",
  heading: "Getting Started"
})
log(section.content)

Rsync

File synchronization.

Parameters

Parameter	Type	Required	Default	Description
host	string	Yes	-	Remote host
username	string	Yes	-	SSH username
privatekeypath	string	No	-	Path to private key
localpath	string	Yes	-	Local path
remotepath	string	Yes	-	Remote path
direction	string	Yes	-	push or pull
archive	bool	No	false	Archive mode (-a)
delete	bool	No	false	Delete extraneous files
exclude	string	No	-	Comma-separated exclusion patterns

Examples

Basic Sync

name: rsync-basic
tasks:
  - name: sync
    rsync:
      host: server.example.com
      username: deploy
      privatekeypath: ~/.ssh/id_rsa
      localpath: ./dist/
      remotepath: /var/www/html/
      direction: push
      archive: true

Sync with Exclusions

name: rsync-exclude
tasks:
  - name: sync-code
    rsync:
      host: server.example.com
      username: deploy
      privatekeypath: ~/.ssh/id_rsa
      localpath: ./project/
      remotepath: /app/
      direction: push
      archive: true
      delete: true
      exclude: "node_modules,.git,.env"

SCP

Secure file transfer.

Parameters

Parameter	Type	Required	Default	Description
host	string	Yes	-	SSH server (host:port)
username	string	Yes	-	SSH username
privatekeypath	string	No	-	Path to private key
localpath	string	Yes	-	Local file/directory path
remotepath	string	Yes	-	Remote file/directory path
direction	string	Yes	-	upload or download
recursive	bool	No	false	Recursive transfer
preservemode	bool	No	false	Preserve file permissions

Examples

Upload File

name: scp-upload
tasks:
  - name: upload
    scp:
      host: server.example.com:22
      username: deploy
      privatekeypath: ~/.ssh/id_rsa
      localpath: ./dist/app.tar.gz
      remotepath: /opt/deploy/app.tar.gz
      direction: upload

Download File

name: scp-download
tasks:
  - name: download
    scp:
      host: server.example.com:22
      username: backup
      privatekeypath: ~/.ssh/backup_key
      localpath: ./backups/db_backup.sql
      remotepath: /backups/latest.sql
      direction: download

S3

S3-compatible object storage operations. Works with AWS S3, MinIO, DigitalOcean Spaces, and other S3-compatible services.

Parameters

Parameter	Type	Required	Default	Description
command	string	Yes	-	Operation: upload, download, list, delete
s3url	string	Yes	-	S3 endpoint URL
bucketname	string	Yes	-	Bucket name
credentials.accesskey	string	Yes	-	Access key ID
credentials.secretkey	string	Yes	-	Secret access key
credentials.secure	bool	No	false	Use HTTPS
dir	string	No	-	Base directory for relative paths
commandparams	object	Yes	-	Command-specific parameters

Command Parameters

Upload

Parameter	Type	Description
files[].path	string	Local file or directory path
files[].destination	string	S3 object key/prefix

Download

Parameter	Type	Description
files[].object	string	S3 object key
files[].path	string	Local destination path

List

Parameter	Type	Description
files[].path	string	Prefix to list
files[].setvar	string	Variable to store results

Delete

Parameter	Type	Description
files[].path	string	S3 object key or prefix
files[].recursive	bool	Delete all objects with prefix

Examples

Upload Files

name: s3-upload
vars:
  s3_access_key: "{{env.AWS_ACCESS_KEY_ID}}"
  s3_secret_key: "{{env.AWS_SECRET_ACCESS_KEY}}"
tasks:
  - name: upload-artifacts
    s3:
      command: upload
      s3url: s3.amazonaws.com
      bucketname: my-bucket
      credentials:
        accesskey: "{{s3_access_key}}"
        secretkey: "{{s3_secret_key}}"
        secure: true
      commandparams:
        files:
          - path: ./dist/app.tar.gz
            destination: releases/v1.0.0/app.tar.gz
          - path: ./dist/checksums.txt
            destination: releases/v1.0.0/checksums.txt

Upload Directory

name: s3-upload-dir
tasks:
  - name: upload-assets
    s3:
      command: upload
      s3url: s3.amazonaws.com
      bucketname: static-assets
      credentials:
        accesskey: "{{s3_access_key}}"
        secretkey: "{{s3_secret_key}}"
        secure: true
      commandparams:
        files:
          - path: ./public
            destination: website/assets/

Download Files

name: s3-download
tasks:
  - name: get-config
    s3:
      command: download
      s3url: s3.amazonaws.com
      bucketname: config-bucket
      credentials:
        accesskey: "{{s3_access_key}}"
        secretkey: "{{s3_secret_key}}"
        secure: true
      commandparams:
        files:
          - object: config/production.yaml
            path: ./config/app.yaml
          - object: secrets/keys.json
            path: ./secrets/keys.json

List Objects

name: s3-list
tasks:
  - name: list-releases
    s3:
      command: list
      s3url: s3.amazonaws.com
      bucketname: my-bucket
      credentials:
        accesskey: "{{s3_access_key}}"
        secretkey: "{{s3_secret_key}}"
        secure: true
      commandparams:
        files:
          - path: releases/
            setvar: release_files

Delete Objects

name: s3-delete
tasks:
  - name: cleanup-old
    s3:
      command: delete
      s3url: s3.amazonaws.com
      bucketname: my-bucket
      credentials:
        accesskey: "{{s3_access_key}}"
        secretkey: "{{s3_secret_key}}"
        secure: true
      commandparams:
        files:
          - path: temp/
            recursive: true
          - path: old-release.tar.gz

MinIO Example

name: s3-minio
tasks:
  - name: upload-to-minio
    s3:
      command: upload
      s3url: minio.local:9000
      bucketname: data
      credentials:
        accesskey: minioadmin
        secretkey: minioadmin
        secure: false
      commandparams:
        files:
          - path: ./backup.sql
            destination: backups/db-backup.sql

Namespace: s3

S3-compatible storage functions.

Functions

Function	Description
s3.upload()	Upload files to S3 bucket
s3.download()	Download files from S3 bucket
s3.list()	List objects in S3 bucket
s3.delete()	Delete objects from S3 bucket

Common Parameters

Parameter	Type	Default	Description
s3Url	string	-	S3 endpoint URL
bucketName	string	-	Bucket name
accessKey	string	-	Access key ID
secretKey	string	-	Secret access key
region	string	-	AWS region
secure	bool	true	Use HTTPS
insecureSkipVerify	bool	false	Skip TLS verification

Upload/Download Parameters

Parameter	Type	Default	Description
localPath	string	-	Local file path
remotePath	string	-	S3 object key
contentType	string	-	Content-Type header

List Parameters

Parameter	Type	Default	Description
prefix	string	-	Filter by prefix
recursive	bool	false	List recursively

Examples

// Upload file
s3.upload({
  s3Url: "s3.amazonaws.com",
  bucketName: "my-bucket",
  accessKey: env.S3_ACCESS_KEY,
  secretKey: env.S3_SECRET_KEY,
  localPath: "./dist/app.tar.gz",
  remotePath: "releases/v1.0.0/app.tar.gz",
  contentType: "application/gzip"
})

// Download file
s3.download({
  s3Url: "s3.amazonaws.com",
  bucketName: "config-bucket",
  accessKey: env.S3_ACCESS_KEY,
  secretKey: env.S3_SECRET_KEY,
  localPath: "./config.yaml",
  remotePath: "config/production.yaml"
})

// List objects
let files = s3.list({
  s3Url: "s3.amazonaws.com",
  bucketName: "my-bucket",
  accessKey: env.S3_ACCESS_KEY,
  secretKey: env.S3_SECRET_KEY,
  prefix: "releases/",
  recursive: true
})
log("Found", files.objects.length, "files")

// Delete object
s3.delete({
  s3Url: "s3.amazonaws.com",
  bucketName: "my-bucket",
  accessKey: env.S3_ACCESS_KEY,
  secretKey: env.S3_SECRET_KEY,
  remotePath: "temp/old-file.txt"
})

// MinIO (self-hosted S3)
s3.upload({
  s3Url: "minio.local:9000",
  bucketName: "data",
  accessKey: "minioadmin",
  secretKey: "minioadmin",
  secure: false,
  localPath: "./backup.sql",
  remotePath: "backups/db-backup.sql"
})

AzureBlob

Azure Blob Storage operations with parallel processing and incremental sync.

Parameters

Parameter	Type	Required	Default	Description
connection_string	string	Yes	-	Azure Storage connection string
commands	list	Yes	-	List of operations to perform
max_retries	int	No	3	Maximum retry attempts
retry_delay	string	No	1s	Delay between retries
max_workers	int	No	4	Parallel workers
incremental	bool	No	false	Only sync changed files
continue_on_error	bool	No	true	Continue on individual file errors
sync_mode	string	No	fast	Sync mode: fast, full
operation_log	string	No	-	Path to write operation log

Command Actions

Action	Description
upload	Upload files to blob storage
download	Download blobs to local filesystem
list	List blobs in container
delete	Delete blobs

Examples

Upload Files

name: azureblob-upload
vars:
  azure_conn: "{{env.AZURE_STORAGE_CONNECTION_STRING}}"
tasks:
  - name: upload-assets
    azureblob:
      connection_string: "{{azure_conn}}"
      commands:
        - action: upload
          container: static-assets
          source: ./dist
          destination: v1.0.0/
          recursive: true
          skip:
            - "*.map"
            - ".DS_Store"

Download with Skip Patterns

name: azureblob-download
tasks:
  - name: sync-configs
    azureblob:
      connection_string: "{{azure_conn}}"
      incremental: true
      commands:
        - action: download
          container: configs
          prefix: production/
          destination: ./config
          skip:
            - "*.bak"
            - "*.old"

List Blobs

name: azureblob-list
tasks:
  - name: list-backups
    azureblob:
      connection_string: "{{azure_conn}}"
      commands:
        - action: list
          container: backups
          prefix: db/

Delete with Prefix

name: azureblob-delete
tasks:
  - name: cleanup
    azureblob:
      connection_string: "{{azure_conn}}"
      commands:
        - action: delete
          container: temp
          prefix: staging/

Parallel Upload with Logging

name: azureblob-parallel
tasks:
  - name: bulk-upload
    azureblob:
      connection_string: "{{azure_conn}}"
      max_workers: 8
      max_retries: 5
      retry_delay: 2s
      continue_on_error: true
      operation_log: ./upload-log.json
      commands:
        - action: upload
          container: data-lake
          source: ./large-dataset
          destination: imports/{{date | date: "%Y-%m-%d"}}/
          recursive: true

Namespace: azureblob

Azure Blob storage functions.

Functions

Function	Description
azureblob.upload()	Upload files to Azure Blob container
azureblob.download()	Download blobs from Azure Blob container
azureblob.list()	List blobs in container
azureblob.delete()	Delete blobs from container

Common Parameters

Parameter	Type	Default	Description
connectionString	string	-	Azure Storage connection string
container	string	-	Container name
localPath	string	-	Local file/directory path
remotePath	string	-	Blob path
recursive	bool	false	Recursive upload/download
skipPatterns	array	-	Patterns to skip
continueOnError	bool	true	Continue on individual file errors
syncMode	string	"fast"	Sync mode: fast or full

Examples

// Upload file
azureblob.upload({
  connectionString: env.AZURE_STORAGE_CONNECTION,
  container: "data",
  localPath: "./dist/app.tar.gz",
  remotePath: "releases/v1.0.0/app.tar.gz"
})

// Upload directory recursively
azureblob.upload({
  connectionString: env.AZURE_STORAGE_CONNECTION,
  container: "assets",
  localPath: "./public",
  remotePath: "website/",
  recursive: true,
  skipPatterns: ["*.tmp", ".git/**"]
})

// Download file
azureblob.download({
  connectionString: env.AZURE_STORAGE_CONNECTION,
  container: "config",
  localPath: "./config.yaml",
  remotePath: "production/config.yaml"
})

// Download directory
azureblob.download({
  connectionString: env.AZURE_STORAGE_CONNECTION,
  container: "backups",
  localPath: "./restore",
  remotePath: "db/",
  recursive: true,
  continueOnError: true
})