Skip to content
Veritas AI

Ingestion

Every document flows through the same state machine: parse → chunk → embed → index, with automatic retry and a quarantine branch for files that repeatedly fail.

Parsing

Parsers are selected by file extension. PDFs are parsed layout-aware — heading hierarchy, tables, and reading order survive into chunk metadata. Pages without a text layer are routed through OCR automatically; pages with one are never re-OCR'd.

Lifecycle & retries

Indexing failures retry with exponential backoff up to a configurable limit. A document that exhausts its retries moves to quarantine with the failure reason attached — it never blocks the rest of the batch.

bash
# Scan a connected source; unchanged files are skipped by content hash
curl -X POST https://api.veritas.example.com/v1/ingest/scan \
  -H "Authorization: Bearer $API_KEY" \
  -d '{"source": "s3://corpus-bucket/policies/"}'

Keeping indexes in sync

Chunks are written to the vector store and the BM25 sidecar through a single write path — the two indexes cannot drift. Cache invalidation events fan out on the same write, so stale cached answers die with the old chunks.

Practical guidance

  • Deduplicate by content hash, not filename — renamed copies are common
  • Keep page numbers: citations depend on them
  • Watch the quarantine queue in the dashboard; a spike almost always means a new document template entered your corpus
PreviousQuickstartNext Chunking