Build text search index

Turn segment records into search postings, per-file summaries and an index manifest.

When to use it

Build searchable postings and file summaries from extracted text segments. The manifest describes the indexed set. A later document replacement or failed extraction needs to be reconciled before treating an old index as current.

Add and configure

Add Build text search index from the workflow step picker.
Give the step a recognisable name and connect its required inputs.
Set the options below. Nested names identify fields within an object or a list item.
Validate the recipe, inspect the test result, then run it on the intended data.

The step identifier used in the editor’s code view is text_index.

Settings

Setting	Accepted value	Required	Default	What it controls
Emit File Records (`emit_file_records`)	boolean	No	true	Emit one search_file row per file.
Emit Manifest (`emit_manifest`)	boolean	No	true	Emit the search_index_meta manifest.
File Key Field (`file_key_field`)	string	No	"object_key"	Segment field identifying the source file.
Max Positions Per Segment (`max_positions_per_segment`)	integer; minimum 1	No	50	Offset-pair cap per (term, segment); match_count stays exact.
Max Segments Per Posting (`max_segments_per_posting`)	integer; minimum 1	No	20	Segment-detail cap per posting; segment_count stays exact.
Max Terms Per File (`max_terms_per_file`)	integer; minimum 1	No	50000	Vocabulary cap per file: once reached, NEW terms stop being admitted (existing terms keep counting) and the file's search_file row carries terms_truncated=true.
Max Token Length (`max_token_length`)	integer; minimum 1; maximum 512	No	64	Drop terms longer than this.
Min Token Length (`min_token_length`)	integer; minimum 1	No	2	Drop terms shorter than this.
Page Field (`page_field`)	string	No	"page_number"	Segment field holding the page number (optional).
Record Type (`record_type`)	string	No	"search_index"	record_type column stamp for ALL emitted rows (read by the runner's generic persister from the raw step config). The payload-level record_type discriminates kinds within the step: search_posting \| search_file \| search_index_meta.
Segment Index Field (`segment_index_field`)	string	No	"index"	Segment field holding the segment's ordinal within its file.
Segment Record Type (`segment_record_type`)	string	No	"document_segment"	Advertised in the manifest: the record_type a viewer queries for the file content this index was built over.
Snippet Chars (`snippet_chars`)	integer; minimum 16; maximum 2000	No	160	Snippet window size.
Stop Words (`stop_words`)	list of string	No	Not specified	Terms to exclude (normalized with the same analyzer rules).
Text Field (`text_field`)	string	No	"text"	Segment field holding the raw text.

Inputs and outputs

Direction	Port	Content	Required	Multiple records
Inputs	`in`	records	Yes	Yes
Outputs	`out`	records	Yes	Yes

Effects, testing, and recovery

Transforms or passes records without a declared external side effect.

The declared preview behaviour is to execute this step. Reads, processing, or model usage can therefore be real. Read dry runs before using a preview with live connections.

No duplicate-prevention key is declared for this step. The step declares safe replay for its supported recovery path.

Example: Index two segments of one file (postings + file row; no manifest)

Step settings:

{
  "emit_manifest": false
}

Input records:

[
  {
    "index": 0,
    "object_key": "docs/a.txt",
    "page_number": 1,
    "text": "alpha beta"
  },
  {
    "index": 1,
    "object_key": "docs/a.txt",
    "page_number": 1,
    "text": "beta gamma"
  }
]

Expected output for this example:

[
  {
    "content_hash": null,
    "document_id": null,
    "file_name": "a.txt",
    "index_version": 1,
    "match_count": 1,
    "object_key": "docs/a.txt",
    "record_type": "search_posting",
    "segment_count": 1,
    "segments": [
      {
        "index": 0,
        "page_number": 1,
        "positions": [
          [
            0,
            5
          ]
        ],
        "positions_truncated": false,
        "snippet": "alpha beta",
        "snippet_start": 0
      }
    ],
    "segments_truncated": false,
    "term": "alpha"
  },
  {
    "content_hash": null,
    "document_id": null,
    "file_name": "a.txt",
    "index_version": 1,
    "match_count": 2,
    "object_key": "docs/a.txt",
    "record_type": "search_posting",
    "segment_count": 2,
    "segments": [
      {
        "index": 0,
        "page_number": 1,
        "positions": [
          [
            6,
            10
          ]
        ],
        "positions_truncated": false,
        "snippet": "alpha beta",
        "snippet_start": 0
      },
      {
        "index": 1,
        "page_number": 1,
        "positions": [
          [
            0,
            4
          ]
        ],
        "positions_truncated": false,
        "snippet": "beta gamma",
        "snippet_start": 0
      }
    ],
    "segments_truncated": false,
    "term": "beta"
  },
  {
    "content_hash": null,
    "document_id": null,
    "file_name": "a.txt",
    "index_version": 1,
    "match_count": 1,
    "object_key": "docs/a.txt",
    "record_type": "search_posting",
    "segment_count": 1,
    "segments": [
      {
        "index": 1,
        "page_number": 1,
        "positions": [
          [
            5,
            10
          ]
        ],
        "positions_truncated": false,
        "snippet": "beta gamma",
        "snippet_start": 0
      }
    ],
    "segments_truncated": false,
    "term": "gamma"
  },
  {
    "content_hash": null,
    "document_id": null,
    "error": null,
    "file_name": "a.txt",
    "index_version": 1,
    "indexed": true,
    "object_key": "docs/a.txt",
    "parse_degraded": false,
    "parse_fallbacks": 0,
    "parse_routes": {},
    "record_type": "search_file",
    "segment_count": 2,
    "term_count": 3,
    "terms_truncated": false,
    "token_count": 4
  }
]

Example: A failed extraction becomes an unindexed file row