Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Retrieval

Biblicus treats retrieval as a reproducible, explicit pipeline stage that transforms a corpus into structured evidence. Retrieval is separated from extraction and context shaping so each can be evaluated independently and swapped without rewriting ingestion.

Retrieval concepts

How retrieval snapshots work

  1. Ingest raw items into a corpus.
  2. Build an extraction snapshot to produce text artifacts.
  3. Build a retrieval snapshot with a backend, referencing the extraction snapshot.
  4. Query the run to return evidence.

Retrieval runs are stored under:

retrieval/<backend_id>/<snapshot_id>/

A minimal run you can execute

This walkthrough uses the full text search backend and produces evidence you can inspect immediately.

rm -rf corpora/retrieval_demo
python -m biblicus init corpora/retrieval_demo
printf "alpha beta\n" > /tmp/retrieval-alpha.txt
printf "beta gamma\n" > /tmp/retrieval-beta.txt
python -m biblicus ingest --corpus corpora/retrieval_demo /tmp/retrieval-alpha.txt
python -m biblicus ingest --corpus corpora/retrieval_demo /tmp/retrieval-beta.txt

python -m biblicus extract build --corpus corpora/retrieval_demo --stage pass-through-text
python -m biblicus build --corpus corpora/retrieval_demo --backend sqlite-full-text-search
python -m biblicus query --corpus corpora/retrieval_demo --query "beta"

The query output is structured evidence with identifiers and scores. That evidence is the primary output for evaluation and downstream context packing.

Backends

See docs/backends/index.md for backend selection and configuration.

Choosing a backend

Start with the simplest backend that answers your question:

You can compare them with the same dataset and budget using the retrieval evaluation workflow.

Evaluation

Retrieval runs are evaluated against datasets with explicit budgets. See docs/retrieval-evaluation.md for the dataset format and workflow, docs/feature-index.md for the behavior specifications, and docs/context-pack.md for how evidence feeds into context packs.

Evidence inspection workflow

When you want to understand a result end to end:

  1. Query the backend and save the output.
  2. Inspect the top evidence items and their scores.
  3. Trace each evidence item_id back to the corpus for context.

Example:

python -m biblicus query --corpus corpora/demo --query "beta" > /tmp/retrieval_output.json
python -c "import json; data=json.load(open('/tmp/retrieval_output.json')); print(data['evidence'][:2])"
python -m biblicus show --corpus corpora/demo ITEM_ID

Saving evidence for later analysis

Evidence output is stable JSON. Save it alongside your experiments so you can compare runs later:

python -m biblicus query --corpus corpora/demo --query "beta" > artifacts/retrieval/beta.json

Record the snapshot identifier and budget values in the same folder so you can reproduce the query.

Labs and demos

When you want a repeatable example with bundled data, use the retrieval evaluation lab:

python scripts/retrieval_evaluation_lab.py --corpus corpora/retrieval_eval_lab --force

The lab builds a tiny corpus, runs extraction, builds a retrieval snapshot, and evaluates it. It prints the dataset path and evaluation output so you can open the JavaScript Object Notation directly.

Reproducibility checklist

Use these habits when you want repeatable retrieval experiments:

Why the separation matters

Keeping extraction and retrieval distinct makes it possible to:

Retrieval quality

For retrieval quality upgrades, see docs/retrieval-quality.md.