Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Retrieval Backends

Biblicus provides pluggable retrieval backends that implement different search and ranking strategies. Each backend defines how evidence is retrieved from your corpus.

Retrieval Backends

Available Backends

scan

Naive full-scan backend that searches all text items at query time without pre-built indexes.

Production-ready full-text search using SQLite FTS5 with BM25 ranking.

tf-vector

Deterministic term-frequency vector retrieval (vector space model baseline) with cosine similarity.

embedding-index-inmemory

Embedding-based retrieval with an in-memory exact cosine similarity index.

embedding-index-file

Embedding-based retrieval with a file-backed exact cosine similarity index.

Quick Start

Installation

All backends are included with the base Biblicus installation:

pip install biblicus

Basic Usage

Command Line

# Initialize corpus
biblicus init my-corpus

# Ingest documents
biblicus ingest my-corpus document.pdf

# Extract text
biblicus extract my-corpus --extractor pdf-text

# Build retrieval snapshot with a backend
biblicus build my-corpus --backend sqlite-full-text-search

# Query the run
biblicus query my-corpus --query "search terms"

See docs/retrieval.md for a step-by-step retrieval walkthrough.

Python API

from biblicus import Corpus
from biblicus.backends import get_backend

# Load corpus
corpus = Corpus.from_directory("my-corpus")

# Get backend
backend = get_backend("sqlite-full-text-search")

# Build run
run = backend.build_run(
    corpus,
    configuration_name="My search index",
    config={}
)

# Query
result = backend.query(
    corpus,
    run=run,
    query_text="search terms",
    budget={"max_total_items": 10}
)

See docs/retrieval-evaluation.md for evaluation workflows and dataset formats.

Choosing a Backend

Use Case Recommended Backend Notes
Development & testing scan No index to build, immediate results
Small corpora (<1000 items) scan Fast enough without indexing overhead
Production applications sqlite-full-text-search Fast queries with BM25 ranking
Large corpora (>10,000 items) sqlite-full-text-search Essential for performance
Baseline comparisons scan Simple reference implementation
Term-frequency vector baseline tf-vector Deterministic cosine similarity
Embedding retrieval (in-memory) embedding-index-inmemory Exact cosine similarity
Embedding retrieval (file-backed) embedding-index-file Exact cosine similarity, memory-mapped

Reproducibility checklist

Common pitfalls

Performance Comparison

Scan Backend

Example: 1000-item corpus, 5-10 second query time

SQLite Full-Text Search Backend

Example: 10,000-item corpus, <100ms query time after indexing

Common Patterns

Development Workflow

Use scan backend during development for immediate feedback:

biblicus build my-corpus --backend scan
biblicus query my-corpus --query "test"

Production Deployment

Build a persistent index with sqlite-full-text-search:

biblicus build my-corpus --backend sqlite-full-text-search \
  --config chunk_size=800 \
  --config chunk_overlap=200

Baseline Comparison

Compare backends using the same corpus:

# Build with both backends
biblicus build my-corpus --backend scan --configuration scan-baseline
biblicus build my-corpus --backend sqlite-full-text-search --configuration fts-index

# Query both
biblicus query my-corpus --run scan:RUN_ID --query "test"
biblicus query my-corpus --run sqlite-full-text-search:RUN_ID --query "test"

Using Extracted Text

All backends support extraction snapshots for non-text content:

# Extract text from PDFs
biblicus extract my-corpus --extractor pdf-text

# Build backend with extraction snapshot
biblicus build my-corpus --backend sqlite-full-text-search \
  --config extraction_snapshot=pdf-text:EXTRACTION_SNAPSHOT_ID

Backend Configuration

Common Configuration Options

All backends support these configuration options:

Option Type Default Description
snippet_characters int 400 Maximum characters in evidence snippets
extraction_snapshot str None Extraction run reference (extractor_id:snapshot_id)

Backend-Specific Options

Scan Backend

No additional configuration options.

SQLite Full-Text Search Backend

Option Type Default Description
chunk_size int 800 Maximum characters per chunk
chunk_overlap int 200 Overlap characters between chunks

Architecture

Backend Interface

All backends implement the RetrievalBackend interface:

class RetrievalBackend:
    backend_id: str

    def build_run(self, corpus, *, configuration_name, config) -> RetrievalRun:
        """Build a retrieval snapshot (may create artifacts)."""

    def query(self, corpus, *, run, query_text, budget) -> RetrievalResult:
        """Query the run and return evidence."""

Evidence Model

All backends return structured Evidence objects:

class Evidence:
    item_id: str                  # Corpus item identifier
    source_uri: Optional[str]     # Original source URI
    media_type: str               # MIME type
    score: float                  # Relevance score
    rank: int                     # Result rank
    text: str                     # Evidence snippet
    span_start: Optional[int]     # Span start offset
    span_end: Optional[int]       # Span end offset
    stage: str                    # Processing stage
    configuration_id: str                # Configuration identifier
    snapshot_id: str                   # Run identifier
    hash: str                     # Content hash

Implementing Custom Backends

To implement a custom backend:

  1. Subclass RetrievalBackend
  2. Implement build_run() and query() methods
  3. Register in biblicus.backends.available_backends
  4. Add BDD specifications with 100% coverage

See backends.md for implementation details.

See Also