Skip to content
Talk to our solutions team

Files and transfer

Filesystem operations, archives, and moving bytes between hosts and object stores. Every parameter with its type, default and description, plus worked examples.

Read and write files.

ParameterTypeRequiredDefaultDescription
operationstringYes-Operation: read or write
filestringYes-File path
contentstringNo-Content to write (for write operation)
templatestringNo-Liquid template to render and write
appendboolNofalseAppend to file instead of overwrite
setvarstringNo-Variable to store file content (for read)
name: read-file
tasks:
- name: load-config
file:
operation: read
file: /config/settings.json
setvar: settings
name: write-file
tasks:
- name: save-result
file:
operation: write
file: /output/result.txt
content: "Hello, World!"
name: write-template
vars:
host: localhost
port: 8080
tasks:
- name: generate-config
file:
operation: write
file: /output/config.yaml
template: |
server:
host: {{host}}
port: {{port}}

File system operations.

FunctionDescription
file.read()Read file contents
file.write()Write content to file
file.exists()Check if file/directory exists
file.remove()Delete file or directory
file.copy()Copy file
file.move()Move/rename file
file.mkdir()Create directory
ParameterTypeDefaultDescription
pathstring-File path to read (required)
ParameterTypeDefaultDescription
pathstring-File path to write (required)
contentstring-Content to write (required)
appendboolfalseAppend instead of overwrite
createbooltrueCreate file if not exists
ParameterTypeDefaultDescription
sourcestring-Source file path (required)
destinationstring-Destination file path (required)
overwriteboolfalseOverwrite if exists
// Read file content
let result = file.read({ path: "/config/settings.json" })
if (result.success) {
log("Content:", result.content)
}
// Write file
file.write({
path: "/output/result.txt",
content: "Hello, World!",
create: true
})
// Append to file
file.write({
path: "/logs/app.log",
content: new Date() + " - Log entry\n",
append: true
})
// Check if file exists
let result = file.exists({ path: "/config/app.yaml" })
if (result.exists) {
log("File exists, isDir:", result.isDir)
}
// Copy file
file.copy({
source: "/source/file.txt",
destination: "/backup/file.txt",
overwrite: true
})
// Move/rename file
file.move({
source: "/old/path.txt",
destination: "/new/path.txt"
})
// Create directory
file.mkdir({
path: "/data/output/reports",
recursive: true
})
// Remove file or directory
file.remove({
path: "/temp/cache",
recursive: true
})

Find files matching glob patterns.

ParameterTypeRequiredDefaultDescription
patternstringYes-Glob pattern (e.g., **/*.go)
pathstringNo.Base directory to search
excludelistNo-Patterns to exclude
filesOnlyboolNofalseReturn only files
dirsOnlyboolNofalseReturn only directories
setvarstringYes-Variable to store matches
name: find-go-files
tasks:
- name: search
glob:
pattern: "**/*.go"
path: /project/src
setvar: go_files
name: find-js-files
tasks:
- name: search
glob:
pattern: "**/*.js"
path: /app
exclude:
- "**/node_modules/**"
- "**/*.test.js"
setvar: source_files

File pattern matching functions.

FunctionDescription
glob.find()Find files matching glob pattern
ParameterTypeDefaultDescription
patternstring-Glob pattern (e.g., **/*.go)
pathstring"."Base directory to search
excludearray-Patterns to exclude
filesOnlyboolfalseReturn only files
dirsOnlyboolfalseReturn only directories
// Find all Go files
let files = glob.find({
pattern: "**/*.go",
path: "/app",
exclude: ["**/vendor/**"]
})
log("Found", files.matches.length, "Go files")
// Find JavaScript files, excluding tests
let jsFiles = glob.find({
pattern: "**/*.js",
path: "/app/src",
exclude: ["**/*.test.js", "**/node_modules/**"],
filesOnly: true
})
// Find all directories
let dirs = glob.find({
pattern: "**/",
path: "/app",
dirsOnly: true
})

Compress and extract ZIP, TAR, TAR.GZ, and GZ archives.

ParameterTypeRequiredDefaultDescription
operationstringYes-Operation: extract or compress
sourcestringYes*-Source file/directory
sourceslistNo-Multiple source paths (for compress)
destinationstringYes-Destination path
formatstringNoauto-detectFormat: zip, tar, tar.gz, gz
stripRootboolNofalseStrip root directory when extracting
setvarstringNo-Variable to store output path
name: extract-zip
tasks:
- name: unpack
archive:
operation: extract
source: /downloads/package.zip
destination: /app
setvar: extracted_path
name: extract-strip
tasks:
- name: extract-node
archive:
operation: extract
source: /downloads/node-v20.0.0-linux-x64.tar.gz
destination: /opt/node
stripRoot: true
name: create-bundle
tasks:
- name: bundle
archive:
operation: compress
format: tar.gz
sources:
- /app/bin
- /app/config
- /app/scripts
destination: /output/app-bundle.tar.gz

Archive extraction and compression functions.

FunctionDescription
archive.extract()Extract archive file
archive.compress()Create archive from files
ParameterTypeDefaultDescription
sourcestring-Path to archive file
destinationstring-Extraction destination
formatstringauto-detectArchive format
stripRootboolfalseStrip root directory
ParameterTypeDefaultDescription
sourcestring-Single source path
sourcesarray-Multiple source paths
destinationstring-Output archive path
formatstringauto-detectArchive format
// Extract archive
let result = archive.extract({
source: "/downloads/package.tar.gz",
destination: "/app",
stripRoot: true
})
log("Extracted to:", result.path)
// Create tar.gz archive
archive.compress({
sources: ["/app/bin", "/app/config"],
destination: "/output/app-bundle.tar.gz",
format: "tar.gz"
})
// Create zip from single directory
archive.compress({
source: "/app/dist",
destination: "/releases/app-v1.0.0.zip"
})

Extract structured content from documents (Markdown, HTML, etc.) into workflow variables.

ParameterTypeRequiredDefaultDescription
formatstringNomarkdownInput format: markdown
contentstringNo*-Inline content to extract from
filestringNo*-File path to read content from
globstringNo*-Glob pattern for multiple files
selectstringNo-CSS/path selector to narrow extraction
languagestringNo-Filter code blocks by language
headingstringNo-Filter sections by heading text
indexintNo-Select a specific match by index
setvarstringNo-Variable to store extracted content
extractionsmapNo-Multiple extractions (variable name to extraction spec)

*At least one of content, file, or glob is required.

name: extract-inline
tasks:
- name: extract-code
extract:
content: |
# README
```yaml
key: value

language: yaml setvar: extracted_yaml

#### Extract from File
```yaml
name: extract-file
tasks:
- name: extract-sections
extract:
file: ./docs/README.md
heading: "Installation"
setvar: install_instructions
name: extract-multi
tasks:
- name: extract-all
extract:
file: ./docs/guide.md
extractions:
install_steps:
heading: "Installation"
config_example:
language: yaml
index: 0
api_reference:
heading: "API"

Content extraction from structured documents.

FunctionDescription
extract.markdown()Extract content from Markdown documents
// Extract code blocks by language
let result = extract.markdown({
content: readmeContent,
language: "yaml"
})
log("Found", result.blocks.length, "YAML blocks")
// Extract section by heading
let section = extract.markdown({
file: "./docs/README.md",
heading: "Getting Started"
})
log(section.content)

File synchronization.

ParameterTypeRequiredDefaultDescription
hoststringYes-Remote host
usernamestringYes-SSH username
privatekeypathstringNo-Path to private key
localpathstringYes-Local path
remotepathstringYes-Remote path
directionstringYes-push or pull
archiveboolNofalseArchive mode (-a)
deleteboolNofalseDelete extraneous files
excludestringNo-Comma-separated exclusion patterns
name: rsync-basic
tasks:
- name: sync
rsync:
host: server.example.com
username: deploy
privatekeypath: ~/.ssh/id_rsa
localpath: ./dist/
remotepath: /var/www/html/
direction: push
archive: true
name: rsync-exclude
tasks:
- name: sync-code
rsync:
host: server.example.com
username: deploy
privatekeypath: ~/.ssh/id_rsa
localpath: ./project/
remotepath: /app/
direction: push
archive: true
delete: true
exclude: "node_modules,.git,.env"

File synchronisation over SSH, via the rsync binary.

FunctionDirection
rsync.push()Local to remote
rsync.pull()Remote to local
rsync.sync()Set by direction

Each returns {success, filesTransferred, totalFiles, bytesSent, bytesReceived, output, exitCode, error}.

ParameterTypeDescription
host, user, portstring, string, intThe remote
keyPathstringSSH private key
localPath, remotePathstringThe two ends
archiveboolArchive mode
recursiveboolRecurse into directories
deleteboolRemove destination files absent from the source
compressboolCompress in transit
checksumboolCompare by checksum rather than size and time
dryRunboolReport without transferring
exclude, includearrayPatterns
excludeFrom, includeFromstringPattern files
partial, partialDirbool, stringKeep partial transfers for resumption
preservePerms, preserveOwner, preserveGroup, preserveTimesboolWhat to carry across
bandwidthLimitintThrottle, in KB/s
jumpHost, jumpUser, jumpPort, jumpKeyPathRoute through a bastion
rsyncPath, sshCommand, extraArgsOverride the invocation
timeoutdurationGive up after this
progress, verboseboolOutput detail
rsync.push({
host: "web-1", user: "deploy", keyPath: "~/.ssh/id_ed25519",
localPath: "./dist/", remotePath: "/srv/app/",
archive: true, delete: true, compress: true
})
// Rehearse first — dryRun reports what would move without moving it
let plan = rsync.push({
host: "web-1", user: "deploy", keyPath: "~/.ssh/id_ed25519",
localPath: "./dist/", remotePath: "/srv/app/",
delete: true, dryRun: true
})
log(plan.totalFiles, "files would transfer")

delete: true removes files at the destination that are not in the source. Run it with dryRun: true once before trusting it against anything you care about.

Secure file transfer.

ParameterTypeRequiredDefaultDescription
hoststringYes-SSH server (host:port)
usernamestringYes-SSH username
privatekeypathstringNo-Path to private key
localpathstringYes-Local file/directory path
remotepathstringYes-Remote file/directory path
directionstringYes-upload or download
recursiveboolNofalseRecursive transfer
preservemodeboolNofalsePreserve file permissions
name: scp-upload
tasks:
- name: upload
scp:
host: server.example.com:22
username: deploy
privatekeypath: ~/.ssh/id_rsa
localpath: ./dist/app.tar.gz
remotepath: /opt/deploy/app.tar.gz
direction: upload
name: scp-download
tasks:
- name: download
scp:
host: server.example.com:22
username: backup
privatekeypath: ~/.ssh/backup_key
localpath: ./backups/db_backup.sql
remotepath: /backups/latest.sql
direction: download

File transfer over SSH, with jump-host and host-key support.

FunctionDirection
scp.upload()Local to remote
scp.download()Remote to local
scp.transfer()Set by direction

Each returns {success, filesTransferred, bytesTransferred, error}.

ParameterTypeDescription
host, userstringThe remote
keyPathstringSSH private key
keyPassphrasestringIf the key is encrypted
passwordstringPassword auth, where a key is not in use
authMethodstringWhich method to use
localPath, remotePathstringThe two ends
recursiveboolTransfer a directory
preserveModeboolCarry file modes across
strictHostKeyCheckingboolVerify the host key
knownHostsPathstringWhere to verify it against
timeoutdurationGive up after this
scp.upload({
host: "web-1", user: "deploy", keyPath: "~/.ssh/id_ed25519",
localPath: "./app.tar.gz", remotePath: "/tmp/app.tar.gz"
})
scp.download({
host: "web-1", user: "deploy", keyPath: "~/.ssh/id_ed25519",
remotePath: "/var/log/app/error.log", localPath: "./error.log"
})

Leave strictHostKeyChecking on and point knownHostsPath at a file you control. Turning it off to make a transfer work removes the only check that the host is the one you meant.

S3-compatible object storage operations. Works with AWS S3, MinIO, DigitalOcean Spaces, and other S3-compatible services.

ParameterTypeRequiredDefaultDescription
commandstringYes-Operation: upload, download, list, delete
s3urlstringYes-S3 endpoint URL
bucketnamestringYes-Bucket name
credentials.accesskeystringYes-Access key ID
credentials.secretkeystringYes-Secret access key
credentials.secureboolNofalseUse HTTPS
dirstringNo-Base directory for relative paths
commandparamsobjectYes-Command-specific parameters
ParameterTypeDescription
files[].pathstringLocal file or directory path
files[].destinationstringS3 object key/prefix
ParameterTypeDescription
files[].objectstringS3 object key
files[].pathstringLocal destination path
ParameterTypeDescription
files[].pathstringPrefix to list
files[].setvarstringVariable to store results
ParameterTypeDescription
files[].pathstringS3 object key or prefix
files[].recursiveboolDelete all objects with prefix
name: s3-upload
vars:
s3_access_key: "{{env.AWS_ACCESS_KEY_ID}}"
s3_secret_key: "{{env.AWS_SECRET_ACCESS_KEY}}"
tasks:
- name: upload-artifacts
s3:
command: upload
s3url: s3.amazonaws.com
bucketname: my-bucket
credentials:
accesskey: "{{s3_access_key}}"
secretkey: "{{s3_secret_key}}"
secure: true
commandparams:
files:
- path: ./dist/app.tar.gz
destination: releases/v1.0.0/app.tar.gz
- path: ./dist/checksums.txt
destination: releases/v1.0.0/checksums.txt
name: s3-upload-dir
tasks:
- name: upload-assets
s3:
command: upload
s3url: s3.amazonaws.com
bucketname: static-assets
credentials:
accesskey: "{{s3_access_key}}"
secretkey: "{{s3_secret_key}}"
secure: true
commandparams:
files:
- path: ./public
destination: website/assets/
name: s3-download
tasks:
- name: get-config
s3:
command: download
s3url: s3.amazonaws.com
bucketname: config-bucket
credentials:
accesskey: "{{s3_access_key}}"
secretkey: "{{s3_secret_key}}"
secure: true
commandparams:
files:
- object: config/production.yaml
path: ./config/app.yaml
- object: secrets/keys.json
path: ./secrets/keys.json
name: s3-list
tasks:
- name: list-releases
s3:
command: list
s3url: s3.amazonaws.com
bucketname: my-bucket
credentials:
accesskey: "{{s3_access_key}}"
secretkey: "{{s3_secret_key}}"
secure: true
commandparams:
files:
- path: releases/
setvar: release_files
name: s3-delete
tasks:
- name: cleanup-old
s3:
command: delete
s3url: s3.amazonaws.com
bucketname: my-bucket
credentials:
accesskey: "{{s3_access_key}}"
secretkey: "{{s3_secret_key}}"
secure: true
commandparams:
files:
- path: temp/
recursive: true
- path: old-release.tar.gz
name: s3-minio
tasks:
- name: upload-to-minio
s3:
command: upload
s3url: minio.local:9000
bucketname: data
credentials:
accesskey: minioadmin
secretkey: minioadmin
secure: false
commandparams:
files:
- path: ./backup.sql
destination: backups/db-backup.sql

S3-compatible storage functions.

FunctionReturns
s3.upload(){success, uploaded, error}
s3.download(){success, downloaded, error}
s3.list(){success, objects, error} — each object has name, size, modTime, isDir
s3.delete(){success, deleted, error}

Every function takes these.

ParameterTypeDefaultDescription
endpointstring-S3 endpoint host, e.g. s3.amazonaws.com or minio.local:9000
bucketstring-Bucket name
accessKeystring-Access key ID
secretKeystring-Secret access key
secureboolfalseUse HTTPS
insecureSkipVerifyboolfalseSkip TLS verification
FunctionParameterTypeDescription
uploadsourcestringLocal path to send
destinationstringObject key to write
recursiveboolWalk a directory
downloadobjectstringObject key to fetch
pathstringLocal path to write
listprefixstringFilter by key prefix
deletepathstringObject key to remove
recursiveboolRemove a prefix
// Upload a file
s3.upload({
endpoint: "s3.amazonaws.com",
bucket: "my-bucket",
accessKey: accessKey,
secretKey: secretKey,
secure: true,
source: "./dist/app.tar.gz",
destination: "releases/v1.0.0/app.tar.gz"
})
// Upload a directory
s3.upload({
endpoint: "minio.local:9000",
bucket: "backups",
accessKey: accessKey,
secretKey: secretKey,
source: "/srv/data",
destination: "nightly/",
recursive: true
})
// Download an object
s3.download({
endpoint: "s3.amazonaws.com",
bucket: "config-bucket",
accessKey: accessKey,
secretKey: secretKey,
secure: true,
object: "config/production.yaml",
path: "./config.yaml"
})
// List a prefix
let found = s3.list({
endpoint: "s3.amazonaws.com",
bucket: "my-bucket",
accessKey: accessKey,
secretKey: secretKey,
secure: true,
prefix: "releases/"
})
log("Found", found.objects.length, "objects")
// Delete an object
s3.delete({
endpoint: "s3.amazonaws.com",
bucket: "my-bucket",
accessKey: accessKey,
secretKey: secretKey,
secure: true,
path: "temp/old-file.txt"
})

Pull credentials from the Vault namespace rather than literals — these examples name variables so the shape is readable, not because that is where a secret should live.

Azure Blob Storage operations with parallel processing and incremental sync.

ParameterTypeRequiredDefaultDescription
connection_stringstringYes-Azure Storage connection string
commandslistYes-List of operations to perform
max_retriesintNo3Maximum retry attempts
retry_delaystringNo1sDelay between retries
max_workersintNo4Parallel workers
incrementalboolNofalseOnly sync changed files
continue_on_errorboolNotrueContinue on individual file errors
sync_modestringNofastSync mode: fast, full
operation_logstringNo-Path to write operation log
ActionDescription
uploadUpload files to blob storage
downloadDownload blobs to local filesystem
listList blobs in container
deleteDelete blobs
name: azureblob-upload
vars:
azure_conn: "{{env.AZURE_STORAGE_CONNECTION_STRING}}"
tasks:
- name: upload-assets
azureblob:
connection_string: "{{azure_conn}}"
commands:
- action: upload
container: static-assets
source: ./dist
destination: v1.0.0/
recursive: true
skip:
- "*.map"
- ".DS_Store"
name: azureblob-download
tasks:
- name: sync-configs
azureblob:
connection_string: "{{azure_conn}}"
incremental: true
commands:
- action: download
container: configs
prefix: production/
destination: ./config
skip:
- "*.bak"
- "*.old"
name: azureblob-list
tasks:
- name: list-backups
azureblob:
connection_string: "{{azure_conn}}"
commands:
- action: list
container: backups
prefix: db/
name: azureblob-delete
tasks:
- name: cleanup
azureblob:
connection_string: "{{azure_conn}}"
commands:
- action: delete
container: temp
prefix: staging/
name: azureblob-parallel
tasks:
- name: bulk-upload
azureblob:
connection_string: "{{azure_conn}}"
max_workers: 8
max_retries: 5
retry_delay: 2s
continue_on_error: true
operation_log: ./upload-log.json
commands:
- action: upload
container: data-lake
source: ./large-dataset
destination: "imports/{{ date | date: '%Y-%m-%d' }}/"
recursive: true

Azure Blob storage functions.

FunctionDescription
azureblob.upload()Upload files to Azure Blob container
azureblob.download()Download blobs from Azure Blob container
azureblob.list()List blobs in container
azureblob.delete()Delete blobs from container
ParameterTypeDefaultDescription
connectionStringstring-Azure Storage connection string
containerstring-Container name
sourcestring-What to send (upload) — local path
destinationstring-Where to write it — blob path on upload, local path on download
prefixstring-Filter by blob prefix (list, download)
blobNamestring-A single blob to act on
maxWorkersint-Parallel transfers
maxRetriesint-Retries per transfer
incrementalboolfalseSkip blobs already present and unchanged
recursiveboolfalseRecursive upload/download
skipPatternsarray-Patterns to skip
continueOnErrorbooltrueContinue on individual file errors
syncModestring"fast"Sync mode: fast or full
// Upload file
azureblob.upload({
connectionString: env.AZURE_STORAGE_CONNECTION,
container: "data",
source: "./dist/app.tar.gz",
destination: "releases/v1.0.0/app.tar.gz"
})
// Upload directory recursively
azureblob.upload({
connectionString: env.AZURE_STORAGE_CONNECTION,
container: "assets",
source: "./public",
destination: "website/",
recursive: true,
skipPatterns: ["*.tmp", ".git/**"]
})
// Download file
azureblob.download({
connectionString: env.AZURE_STORAGE_CONNECTION,
container: "config",
source: "./config.yaml",
destination: "production/config.yaml"
})
// Download directory
azureblob.download({
connectionString: env.AZURE_STORAGE_CONNECTION,
container: "backups",
source: "./restore",
destination: "db/",
recursive: true,
continueOnError: true
})