# Build text search index

Turn segment records into search postings, per-file summaries and an index manifest.

## When to use it

Build searchable postings and file summaries from extracted text segments. The manifest describes the indexed set. A later document replacement or failed extraction needs to be reconciled before treating an old index as current.

## Add and configure

1. Add **Build text search index** from the workflow step picker.
2. Give the step a recognisable name and connect its required inputs.
3. Set the options below. Nested names identify fields within an object or a list item.
4. Validate the recipe, inspect the test result, then run it on the intended data.

The step identifier used in the editor’s code view is `text_index`.

## Settings

| Setting | Accepted value | Required | Default | What it controls |
| --- | --- | --- | --- | --- |
| Emit File Records (`emit_file_records`) | boolean | No | true | Emit one search_file row per file. |
| Emit Manifest (`emit_manifest`) | boolean | No | true | Emit the search_index_meta manifest. |
| File Key Field (`file_key_field`) | string | No | "object_key" | Segment field identifying the source file. |
| Max Positions Per Segment (`max_positions_per_segment`) | integer; minimum 1 | No | 50 | Offset-pair cap per (term, segment); match_count stays exact. |
| Max Segments Per Posting (`max_segments_per_posting`) | integer; minimum 1 | No | 20 | Segment-detail cap per posting; segment_count stays exact. |
| Max Terms Per File (`max_terms_per_file`) | integer; minimum 1 | No | 50000 | Vocabulary cap per file: once reached, NEW terms stop being admitted (existing terms keep counting) and the file's search_file row carries terms_truncated=true. |
| Max Token Length (`max_token_length`) | integer; minimum 1; maximum 512 | No | 64 | Drop terms longer than this. |
| Min Token Length (`min_token_length`) | integer; minimum 1 | No | 2 | Drop terms shorter than this. |
| Page Field (`page_field`) | string | No | "page_number" | Segment field holding the page number (optional). |
| Record Type (`record_type`) | string | No | "search_index" | record_type column stamp for ALL emitted rows (read by the runner's generic persister from the raw step config). The payload-level record_type discriminates kinds within the step: search_posting \| search_file \| search_index_meta. |
| Segment Index Field (`segment_index_field`) | string | No | "index" | Segment field holding the segment's ordinal within its file. |
| Segment Record Type (`segment_record_type`) | string | No | "document_segment" | Advertised in the manifest: the record_type a viewer queries for the file content this index was built over. |
| Snippet Chars (`snippet_chars`) | integer; minimum 16; maximum 2000 | No | 160 | Snippet window size. |
| Stop Words (`stop_words`) | list of string | No | Not specified | Terms to exclude (normalized with the same analyzer rules). |
| Text Field (`text_field`) | string | No | "text" | Segment field holding the raw text. |

## Inputs and outputs

| Direction | Port | Content | Required | Multiple records |
| --- | --- | --- | --- | --- |
| Inputs | `in` | records | Yes | Yes |
| Outputs | `out` | records | Yes | Yes |

## Effects, testing, and recovery

Transforms or passes records without a declared external side effect.

The declared preview behaviour is to execute this step. Reads, processing, or model usage can therefore be real. Read [dry runs](/docs/workflows/dry-runs) before using a preview with live connections.

No duplicate-prevention key is declared for this step. The step declares safe replay for its supported recovery path.

## Example: Index two segments of one file (postings + file row; no manifest)

Step settings:

```json
{
  "emit_manifest": false
}
```

Input records:

```json
[
  {
    "index": 0,
    "object_key": "docs/a.txt",
    "page_number": 1,
    "text": "alpha beta"
  },
  {
    "index": 1,
    "object_key": "docs/a.txt",
    "page_number": 1,
    "text": "beta gamma"
  }
]
```

Expected output for this example:

```json
[
  {
    "content_hash": null,
    "document_id": null,
    "file_name": "a.txt",
    "index_version": 1,
    "match_count": 1,
    "object_key": "docs/a.txt",
    "record_type": "search_posting",
    "segment_count": 1,
    "segments": [
      {
        "index": 0,
        "page_number": 1,
        "positions": [
          [
            0,
            5
          ]
        ],
        "positions_truncated": false,
        "snippet": "alpha beta",
        "snippet_start": 0
      }
    ],
    "segments_truncated": false,
    "term": "alpha"
  },
  {
    "content_hash": null,
    "document_id": null,
    "file_name": "a.txt",
    "index_version": 1,
    "match_count": 2,
    "object_key": "docs/a.txt",
    "record_type": "search_posting",
    "segment_count": 2,
    "segments": [
      {
        "index": 0,
        "page_number": 1,
        "positions": [
          [
            6,
            10
          ]
        ],
        "positions_truncated": false,
        "snippet": "alpha beta",
        "snippet_start": 0
      },
      {
        "index": 1,
        "page_number": 1,
        "positions": [
          [
            0,
            4
          ]
        ],
        "positions_truncated": false,
        "snippet": "beta gamma",
        "snippet_start": 0
      }
    ],
    "segments_truncated": false,
    "term": "beta"
  },
  {
    "content_hash": null,
    "document_id": null,
    "file_name": "a.txt",
    "index_version": 1,
    "match_count": 1,
    "object_key": "docs/a.txt",
    "record_type": "search_posting",
    "segment_count": 1,
    "segments": [
      {
        "index": 1,
        "page_number": 1,
        "positions": [
          [
            5,
            10
          ]
        ],
        "positions_truncated": false,
        "snippet": "beta gamma",
        "snippet_start": 0
      }
    ],
    "segments_truncated": false,
    "term": "gamma"
  },
  {
    "content_hash": null,
    "document_id": null,
    "error": null,
    "file_name": "a.txt",
    "index_version": 1,
    "indexed": true,
    "object_key": "docs/a.txt",
    "parse_degraded": false,
    "parse_fallbacks": 0,
    "parse_routes": {},
    "record_type": "search_file",
    "segment_count": 2,
    "term_count": 3,
    "terms_truncated": false,
    "token_count": 4
  }
]
```

## Example: A failed extraction becomes an unindexed file row

Step settings:

```json
{
  "emit_manifest": false
}
```

Input records:

```json
[
  {
    "error": "unsupported encryption",
    "object_key": "docs/broken.pdf",
    "record_type": "document_error"
  }
]
```

Expected output for this example:

```json
[
  {
    "content_hash": null,
    "document_id": null,
    "error": "unsupported encryption",
    "file_name": "broken.pdf",
    "index_version": 1,
    "indexed": false,
    "object_key": "docs/broken.pdf",
    "parse_degraded": false,
    "parse_fallbacks": 0,
    "parse_routes": {},
    "record_type": "search_file",
    "segment_count": 0,
    "term_count": 0,
    "terms_truncated": false,
    "token_count": 0
  }
]
```

## Check the result

Inspect the selected run’s status, counts, errors, and output records. An empty collection can be a valid result, and a completed step can contain per-record failures where the step defines that behaviour. Check the next step’s expected record shape before publishing or delivering the result.