Web crawl

Fetch each record's URL and extract clean article markdown.

When to use it

Fetch public web pages and turn readable article content into Markdown. A successful HTTP response can still contain a paywall, login page, or little extractable content. Read the crawl outcome before generating a summary.

Add and configure

Add Web crawl from the workflow step picker.
Give the step a recognisable name and connect its required inputs.
Set the options below. Nested names identify fields within an object or a list item.
Validate the recipe, inspect the test result, then run it on the intended data.

The step identifier used in the editor’s code view is web_crawl.

Settings

Setting	Accepted value	Required	Default	What it controls
Concurrency (`concurrency`)	integer; minimum 1; maximum 32	No	4	Max concurrent fetches for the whole step — the MEMORY bound (multiplied by 'max_source_bytes' it is the step's footprint).
Max Chars (`max_chars`)	integer; minimum 1; maximum 500000	No	50000	Truncate the extracted markdown body to this many characters (a short truncation marker is appended when it trips).
Max Source Bytes (`max_source_bytes`)	integer; minimum 1024; maximum 4194304	No	2097152	Hard cap on the fetched page BEFORE it is decoded or parsed; the stream is abandoned past it and the record degrades to error_code 'too_large'. Multiplied by 'concurrency' this is the step's memory footprint, so raise both together with care.
Per Host Concurrency (`per_host_concurrency`)	integer; minimum 1; maximum 8	No	2	Max concurrent fetches against ONE origin — the politeness bound. Acquired OUTSIDE the global 'concurrency' slot, so a record queued on a busy host holds none of the step's memory capacity. A feed whose articles all share a host no longer arrives as one burst.
reliability (`reliability`)	object or null	No	null	Opt-in reliability profile. Unset (the default) is today's behaviour: the step waits for every record and the runner's own timer is the only bound. Set it (even to '{}', which is all defaults) to give the step its own clock — it stops waiting for records early enough to degrade the unfinished ones and record what it did, instead of being killed mid-step.
Cleanup Budget Seconds (`reliability.cleanup_budget_seconds`)	number; minimum 1.0; maximum 30.0	No	5.0	Total budget for draining the records still in flight and closing the HTTP client once fetching stops. Measured from the moment cleanup begins, so a healthy step does not hand the close its unused fetch budget.
Handoff Seconds (`reliability.handoff_seconds`)	number; minimum 1.0; maximum 15.0	No	3.0	Slack left to the runner's own step timer. With this profile set, the handler stops fetching by 'step timeout - this', so the runner has time to record the outcome instead of killing the step mid-write. It also bounds how long cleanup may borrow.
Timeout Seconds (`timeout_seconds`)	number; minimum 1.0; maximum 300.0	No	15.0	Total deadline for ONE record's page fetch (resolution + connect + whole body). WARNING: the runner ALSO reads this key as the step's wall-clock timeout, so a value sized for a single slow page fails the whole run once the batch outlives it — leave it unset (default) or size it for the whole batch.
Url Field (`url_field`)	string	No	"url"	Record field holding the URL to fetch.

Inputs and outputs

Direction	Port	Content	Required	Multiple records
Inputs	`in`	records	Yes	Yes
Outputs	`out`	records	Yes	Yes

Effects, testing, and recovery

Reads an external source.

The declared preview behaviour is to execute this step. Reads, processing, or model usage can therefore be real. Read dry runs before using a preview with live connections.

No duplicate-prevention key is declared for this step. Do not assume that manually repeating the step is free of repeated effects.

Example: Crawl each search hit's URL into article markdown

Step settings:

{
  "url_field": "url"
}

Input records:

[
  {
    "title": "Rates held",
    "url": "https://news.example.com/2026/09/rates"
  }
]

Example: Polite crawl of one site with its own step clock