# Deduplicate

Drop records whose key values repeat, keeping the first or last occurrence.

## When to use it

Choose the key or keys that identify a duplicate and whether the first or last occurrence should survive. Duplicates are based on those values, not on whether two records merely look similar. Sort first when the occurrence you retain depends on order.

## Add and configure

1. Add **Deduplicate** from the workflow step picker.
2. Give the step a recognisable name and connect its required inputs.
3. Set the options below. Nested names identify fields within an object or a list item.
4. Validate the recipe, inspect the test result, then run it on the intended data.

The step identifier used in the editor’s code view is `dedupe`.

## Settings

| Setting | Accepted value | Required | Default | What it controls |
| --- | --- | --- | --- | --- |
| Keep (`keep`) | "first" / "last" | No | "first" | Which occurrence of a duplicated key survives. Survivors keep their own position, so the output stays in input order either way. |
| Keys (`keys`) | list of string; minimum items 1 | Yes | Not specified | JSONPaths whose values together identify a record (e.g. `["$.url"]`). Two records are duplicates when every key yields the same JSON value. |
| Normalize (`normalize`) | list of "trim" / "lower" / "casefold" / "collapse_whitespace" / "nfkc" / "url" | No | Not specified | Normalizers applied, in order, to every string key value before comparing: `trim`, `lower`, `casefold`, `collapse_whitespace`, `nfkc`, `url`. The records themselves are never changed. |
| On Missing (`on_missing`) | "keep" / "drop" / "group" | No | "keep" | A record where a key path matches nothing: `keep` passes it through (it is nobody's duplicate), `drop` removes it, `group` compares the missing key as one more value, so such records dedupe among themselves. |

## Inputs and outputs

| Direction | Port | Content | Required | Multiple records |
| --- | --- | --- | --- | --- |
| Inputs | `in` | records | Yes | Yes |
| Outputs | `out` | records | Yes | Yes |

## Effects, testing, and recovery

Transforms or passes records without a declared external side effect.

The declared preview behaviour is to execute this step. Reads, processing, or model usage can therefore be real. Read [dry runs](/docs/workflows/dry-runs) before using a preview with live connections.

No duplicate-prevention key is declared for this step. The step declares safe replay for its supported recovery path.

## Example: One record per link, ignoring tracking parameters

Step settings:

```json
{
  "keys": [
    "$.url"
  ],
  "normalize": [
    "url"
  ]
}
```

Input records:

```json
[
  {
    "rank": 1,
    "url": "https://www.example.com/a/?utm_source=feed"
  },
  {
    "rank": 2,
    "url": "http://example.com/a"
  },
  {
    "rank": 3,
    "url": "https://example.com/b"
  }
]
```

Expected output for this example:

```json
[
  {
    "rank": 1,
    "url": "https://www.example.com/a/?utm_source=feed"
  },
  {
    "rank": 3,
    "url": "https://example.com/b"
  }
]
```

## Example: Keep the latest record per id

Step settings:

```json
{
  "keep": "last",
  "keys": [
    "$.id"
  ]
}
```

Input records:

```json
[
  {
    "id": 1,
    "status": "open"
  },
  {
    "id": 2,
    "status": "open"
  },
  {
    "id": 1,
    "status": "closed"
  }
]
```

Expected output for this example:

```json
[
  {
    "id": 2,
    "status": "open"
  },
  {
    "id": 1,
    "status": "closed"
  }
]
```

## Example: Compound key, case- and space-insensitive

Step settings:

```json
{
  "keys": [
    "$.first",
    "$.last"
  ],
  "normalize": [
    "collapse_whitespace",
    "casefold"
  ]
}
```

Input records:

```json
[
  {
    "first": "Ada",
    "last": "Lovelace"
  },
  {
    "first": " ada ",
    "last": "LOVELACE"
  },
  {
    "first": "Ada",
    "last": "Byron"
  }
]
```

Expected output for this example:

```json
[
  {
    "first": "Ada",
    "last": "Lovelace"
  },
  {
    "first": "Ada",
    "last": "Byron"
  }
]
```

## Check the result

Inspect the selected run’s status, counts, errors, and output records. An empty collection can be a valid result, and a completed step can contain per-record failures where the step defines that behaviour. Check the next step’s expected record shape before publishing or delivering the result.