# Web crawl

Fetch each record's URL and extract clean article markdown.

## When to use it

Fetch public web pages and turn readable article content into Markdown. A successful HTTP response can still contain a paywall, login page, or little extractable content. Read the crawl outcome before generating a summary.

## Add and configure

1. Add **Web crawl** from the workflow step picker.
2. Give the step a recognisable name and connect its required inputs.
3. Set the options below. Nested names identify fields within an object or a list item.
4. Validate the recipe, inspect the test result, then run it on the intended data.

The step identifier used in the editor’s code view is `web_crawl`.

## Settings

| Setting | Accepted value | Required | Default | What it controls |
| --- | --- | --- | --- | --- |
| Concurrency (`concurrency`) | integer; minimum 1; maximum 32 | No | 4 | Max concurrent fetches for the whole step — the MEMORY bound (multiplied by 'max_source_bytes' it is the step's footprint). |
| Max Chars (`max_chars`) | integer; minimum 1; maximum 500000 | No | 50000 | Truncate the extracted markdown body to this many characters (a short truncation marker is appended when it trips). |
| Max Source Bytes (`max_source_bytes`) | integer; minimum 1024; maximum 4194304 | No | 2097152 | Hard cap on the fetched page BEFORE it is decoded or parsed; the stream is abandoned past it and the record degrades to error_code 'too_large'. Multiplied by 'concurrency' this is the step's memory footprint, so raise both together with care. |
| Per Host Concurrency (`per_host_concurrency`) | integer; minimum 1; maximum 8 | No | 2 | Max concurrent fetches against ONE origin — the politeness bound. Acquired OUTSIDE the global 'concurrency' slot, so a record queued on a busy host holds none of the step's memory capacity. A feed whose articles all share a host no longer arrives as one burst. |
| reliability (`reliability`) | object or null | No | null | Opt-in reliability profile. Unset (the default) is today's behaviour: the step waits for every record and the runner's own timer is the only bound. Set it (even to '{}', which is all defaults) to give the step its own clock — it stops waiting for records early enough to degrade the unfinished ones and record what it did, instead of being killed mid-step. |
| Cleanup Budget Seconds (`reliability.cleanup_budget_seconds`) | number; minimum 1.0; maximum 30.0 | No | 5.0 | Total budget for draining the records still in flight and closing the HTTP client once fetching stops. Measured from the moment cleanup begins, so a healthy step does not hand the close its unused fetch budget. |
| Handoff Seconds (`reliability.handoff_seconds`) | number; minimum 1.0; maximum 15.0 | No | 3.0 | Slack left to the runner's own step timer. With this profile set, the handler stops fetching by 'step timeout - this', so the runner has time to record the outcome instead of killing the step mid-write. It also bounds how long cleanup may borrow. |
| Timeout Seconds (`timeout_seconds`) | number; minimum 1.0; maximum 300.0 | No | 15.0 | Total deadline for ONE record's page fetch (resolution + connect + whole body). WARNING: the runner ALSO reads this key as the step's wall-clock timeout, so a value sized for a single slow page fails the whole run once the batch outlives it — leave it unset (default) or size it for the whole batch. |
| Url Field (`url_field`) | string | No | "url" | Record field holding the URL to fetch. |

## Inputs and outputs

| Direction | Port | Content | Required | Multiple records |
| --- | --- | --- | --- | --- |
| Inputs | `in` | records | Yes | Yes |
| Outputs | `out` | records | Yes | Yes |

## Effects, testing, and recovery

Reads an external source.

The declared preview behaviour is to execute this step. Reads, processing, or model usage can therefore be real. Read [dry runs](/docs/workflows/dry-runs) before using a preview with live connections.

No duplicate-prevention key is declared for this step. Do not assume that manually repeating the step is free of repeated effects.

## Example: Crawl each search hit's URL into article markdown

Step settings:

```json
{
  "url_field": "url"
}
```

Input records:

```json
[
  {
    "title": "Rates held",
    "url": "https://news.example.com/2026/09/rates"
  }
]
```

## Example: Polite crawl of one site with its own step clock

Step settings:

```json
{
  "max_chars": 20000,
  "per_host_concurrency": 1,
  "reliability": {},
  "url_field": "link"
}
```

Input records:

```json
[
  {
    "link": "https://blog.example.org/posts/launch"
  }
]
```

## Check the result

Inspect the selected run’s status, counts, errors, and output records. An empty collection can be a valid result, and a completed step can contain per-record failures where the step defines that behaviour. Check the next step’s expected record shape before publishing or delivering the result.