Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Scan Backend

Backend ID: scan

Category: Retrieval Backends

Overview

The scan backend is a naive full-scan retrieval implementation that searches all text items at query time without building a persistent index. It provides a simple baseline for retrieval evaluation and is suitable for small corpora or development workflows.

The scan backend tokenizes queries into terms, scores items by term frequency, and returns ranked evidence with snippet extraction. It requires no build time but scales linearly with corpus size.

Installation

The scan backend is included by default with Biblicus:

pip install biblicus

No additional dependencies or setup required.

When to Use

Good Use Cases

Configuration

Config Schema

class ScanRecipeConfig(BaseModel):
    snippet_characters: int = 400       # Maximum characters in snippets
    extraction_snapshot: Optional[str] = None  # Extraction run reference

Configuration Options

Option Type Default Description
snippet_characters int 400 Maximum characters to include in evidence snippets
extraction_snapshot str None Optional extraction snapshot reference (extractor_id:snapshot_id)

Usage

Command Line

Basic Usage

# Build scan run (no artifacts created)
biblicus build my-corpus --backend scan

# Query the run
biblicus query my-corpus --query "search terms"

Custom Configuration

# Larger snippets
biblicus build my-corpus --backend scan \
  --config snippet_characters=800

# With extraction snapshot
biblicus build my-corpus --backend scan \
  --config extraction_snapshot=pdf-text:abc123

Configuration File

backend_id: scan
configuration_name: "Development scan"
config:
  snippet_characters: 400
  extraction_snapshot: null
biblicus build my-corpus --configuration configuration.yml

Python API

from biblicus import Corpus
from biblicus.backends import get_backend
from biblicus.models import QueryBudget

# Load corpus
corpus = Corpus.from_directory("my-corpus")

# Get scan backend
backend = get_backend("scan")

# Build run (no index created)
run = backend.build_run(
    corpus,
    configuration_name="Quick scan",
    config={}
)

# Query
result = backend.query(
    corpus,
    run=run,
    query_text="search terms",
    budget=QueryBudget(max_total_items=10)
)

# Access evidence
for evidence in result.evidence:
    print(f"{evidence.item_id}: {evidence.score}")
    print(f"  {evidence.text[:100]}...")

With Extraction Runs

# Extract text first
extraction_snapshot = corpus.extract_text(
    extractor_id="pdf-text"
)

# Build scan with extraction
run = backend.build_run(
    corpus,
    configuration_name="Scan with extraction",
    config={
        "extraction_snapshot": f"pdf-text:{extraction_snapshot.snapshot_id}"
    }
)

How It Works

Query Processing

  1. Tokenization: Query text is lowercased and split into tokens
  2. Scanning: All corpus items are loaded and scanned sequentially
  3. Scoring: Items are scored by term frequency (count of query tokens in text)
  4. Ranking: Scored items are sorted by score (descending), then by item ID
  5. Snippet Extraction: First match location is found, snippet extracted around it
  6. Budget Application: Top-ranked items are selected according to query budget

Scoring Algorithm

def score_item(item_text, query_tokens):
    """Score = sum of token frequencies in item text."""
    lower_text = item_text.lower()
    return sum(lower_text.count(token) for token in query_tokens)

Snippet Extraction

Performance

Build Time

Query Time

Memory Usage

Disk Usage

Examples

# Initialize and populate corpus
biblicus init demo-corpus
echo "Machine learning applications" > ml.txt
echo "Deep learning neural networks" > dl.txt
biblicus ingest demo-corpus ml.txt dl.txt

# Immediate search (no index build)
biblicus build demo-corpus --backend scan
biblicus query demo-corpus --query "learning"

Baseline Comparison

from biblicus import Corpus
from biblicus.backends import get_backend

corpus = Corpus.from_directory("test-corpus")

# Build with both backends
scan_backend = get_backend("scan")
fts_backend = get_backend("sqlite-full-text-search")

scan_run = scan_backend.build_run(corpus, configuration_name="Scan baseline", config={})
fts_run = fts_backend.build_run(corpus, configuration_name="FTS index", config={})

# Compare results
query = "neural networks"
budget = {"max_total_items": 10}

scan_result = scan_backend.query(corpus, run=scan_run, query_text=query, budget=budget)
fts_result = fts_backend.query(corpus, run=fts_run, query_text=query, budget=budget)

print(f"Scan returned {len(scan_result.evidence)} items")
print(f"FTS returned {len(fts_result.evidence)} items")

Ad-hoc Exploration

# No commitment to index structure
biblicus build corpus --backend scan
biblicus query corpus --query "term1"
biblicus query corpus --query "term2"
biblicus query corpus --query "term3"

# Switch to FTS when ready
biblicus build corpus --backend sqlite-full-text-search

Limitations

Scalability

Ranking Quality

Query Features

When to Upgrade

Consider switching to sqlite-full-text-search when:

Error Handling

Missing Extraction Run

If configured extraction snapshot doesn't exist:

FileNotFoundError: Missing extraction snapshot: pdf-text:abc123

Fix: Verify extraction snapshot ID or run extraction first.

Non-Text Items

Non-text items without extraction snapshot are skipped automatically. No error raised.

Statistics

Build run statistics:

{
  "items": 1000,
  "text_items": 850
}

Query result statistics:

{
  "candidates": 42,
  "returned": 10
}

See Also