Skip to content
Docs

Start here

↑ ↓ navigate · Enter open · Esc closeSearch all 418 articles

Copy files between stores

Configure the file-copy starter and verify its source scope and destination.

Use Copy files between stores to copy a selected object-store prefix into another store. The prepared graph lists source files with read_files, then passes that explicit list to copy. The original files remain in the source.

Configure the template

ParameterWhat to enter
workflow_nameA nonempty name for the new workflow.
source_connection_idThe saved source store connection.
source_catalog_slugThe source connector type; the starter defaults to platform_store.
destination_connection_idThe saved destination store connection.
destination_catalog_slugThe destination connector type; the starter defaults to s3_storage.
prefixOptional source key prefix; blank selects the source's broader accessible scope.

Both connection identities and connector types must match the actual saved stores. A destination connection is not implied by choosing its connector type. Check read permission on the source and write permission on the destination.

Create and test

  1. Choose the template from the workflow gallery.
  2. Supply the required name and store bindings.
  3. Start with a narrow prefix containing a few known files.
  4. Create the workflow and inspect both jobs.
  5. Validate, then start a controlled real run.
  6. Inspect the destination and confirm the expected files and contents.

The starter is enabled with a Manual trigger. Its listing job avoids publishing a separate permanent dataset for the file list. The copy job uses selection: input: only files emitted by the listing job are eligible, and an empty list copies nothing. This differs from automatically listing a broad prefix again inside the destination step.

Check completion and repetition

A copied file is an external destination effect. A workflow output preview does not prove that every byte is present in the destination. Inspect file size and content where the transfer matters.

Before repeating the run, check how the destination handles existing object keys. Do not assume this starter renames collisions or implements a two-way synchronization. It does not delete the originals, remove files absent from the source, or turn a copy into a move.

If no files copy, inspect the prefix, source permissions, and listing output. For failures, inspect partial destination results before retrying.

See Read file list, Copy files, and storage collisions.

Prepared job settings

The starter is enabled and uses a Manual trigger. These settings are supplied by the recipe; review them in the editor before adapting the process. ${…} values below refer to the corresponding template parameter or run variable.

SettingPrepared value
read_files.config.source.kind"connection"
read_files.config.source.connection_id"${source_connection_id}"
read_files.config.source.catalog_slug"${source_catalog_slug}"
read_files.config.prefix"${prefix}"
read_files.config.persist_outputsfalse
copy.config.source.kind"connection"
copy.config.source.connection_id"${source_connection_id}"
copy.config.source.catalog_slug"${source_catalog_slug}"
copy.config.destination.kind"connection"
copy.config.destination.connection_id"${destination_connection_id}"
copy.config.destination.catalog_slug"${destination_catalog_slug}"
copy.config.prefix"${prefix}"
copy.config.selection"input"