Deduplicate
Drop records whose key values repeat, keeping the first or last occurrence.
Drop records whose key values repeat, keeping the first or last occurrence.
When to use it
Choose the key or keys that identify a duplicate and whether the first or last occurrence should survive. Duplicates are based on those values, not on whether two records merely look similar. Sort first when the occurrence you retain depends on order.
Add and configure
- Add Deduplicate from the workflow step picker.
- Give the step a recognisable name and connect its required inputs.
- Set the options below. Nested names identify fields within an object or a list item.
- Validate the recipe, inspect the test result, then run it on the intended data.
The step identifier used in the editor’s code view is dedupe.
Settings
| Setting | Accepted value | Required | Default | What it controls |
|---|---|---|---|---|
Keep (keep) | "first" / "last" | No | "first" | Which occurrence of a duplicated key survives. Survivors keep their own position, so the output stays in input order either way. |
Keys (keys) | list of string; minimum items 1 | Yes | Not specified | JSONPaths whose values together identify a record (e.g. ["$.url"]). Two records are duplicates when every key yields the same JSON value. |
Normalize (normalize) | list of "trim" / "lower" / "casefold" / "collapse_whitespace" / "nfkc" / "url" | No | Not specified | Normalizers applied, in order, to every string key value before comparing: trim, lower, casefold, collapse_whitespace, nfkc, url. The records themselves are never changed. |
On Missing (on_missing) | "keep" / "drop" / "group" | No | "keep" | A record where a key path matches nothing: keep passes it through (it is nobody's duplicate), drop removes it, group compares the missing key as one more value, so such records dedupe among themselves. |
Inputs and outputs
| Direction | Port | Content | Required | Multiple records |
|---|---|---|---|---|
| Inputs | in | records | Yes | Yes |
| Outputs | out | records | Yes | Yes |
Effects, testing, and recovery
Transforms or passes records without a declared external side effect.
The declared preview behaviour is to execute this step. Reads, processing, or model usage can therefore be real. Read dry runs before using a preview with live connections.
No duplicate-prevention key is declared for this step. The step declares safe replay for its supported recovery path.
Example: One record per link, ignoring tracking parameters
Step settings:
{
"keys": [
"$.url"
],
"normalize": [
"url"
]
}
Input records:
[
{
"rank": 1,
"url": "https://www.example.com/a/?utm_source=feed"
},
{
"rank": 2,
"url": "http://example.com/a"
},
{
"rank": 3,
"url": "https://example.com/b"
}
]
Expected output for this example:
[
{
"rank": 1,
"url": "https://www.example.com/a/?utm_source=feed"
},
{
"rank": 3,
"url": "https://example.com/b"
}
]
Example: Keep the latest record per id
Step settings:
{
"keep": "last",
"keys": [
"$.id"
]
}
Input records:
[
{
"id": 1,
"status": "open"
},
{
"id": 2,
"status": "open"
},
{
"id": 1,
"status": "closed"
}
]
Expected output for this example:
[
{
"id": 2,
"status": "open"
},
{
"id": 1,
"status": "closed"
}
]
Example: Compound key, case- and space-insensitive
Step settings:
{
"keys": [
"$.first",
"$.last"
],
"normalize": [
"collapse_whitespace",
"casefold"
]
}
Input records:
[
{
"first": "Ada",
"last": "Lovelace"
},
{
"first": " ada ",
"last": "LOVELACE"
},
{
"first": "Ada",
"last": "Byron"
}
]
Expected output for this example:
[
{
"first": "Ada",
"last": "Lovelace"
},
{
"first": "Ada",
"last": "Byron"
}
]
Check the result
Inspect the selected run’s status, counts, errors, and output records. An empty collection can be a valid result, and a completed step can contain per-record failures where the step defines that behaviour. Check the next step’s expected record shape before publishing or delivering the result.