Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Corpus design

This document records design decisions and outcomes for corpus management and lifecycle hooks in version zero.

The goal is to make corpus management practical for day to day use, while keeping the raw corpus durable and readable as ordinary files on disk.

What exists today

The project already supports:

The decisions below describe how version zero refined and extended these workflows without changing the core principle that the corpus is the source of truth.

Corpus layout

Biblicus owns the on-disk layout of a corpus. Code outside Biblicus should ask a Corpus for these locations instead of joining path segments itself, so the layout can change in one place.

Path under the corpus root Corpus accessor
metadata/catalog.json Corpus.catalog_path (property, already existed)
extracted/<extractor_id>/<snapshot_id>/ Corpus.extraction_snapshot_dir(extractor_id=..., snapshot_id=...)
extracted/pipeline/<snapshot_id>/text/<item_id>.txt Corpus.extracted_text_path(snapshot_id, item_id)
graph/<extractor_id>/<snapshot_id>/ Corpus.graph_snapshot_dir(extractor_id=..., snapshot_id=...)
graph/<extractor_id>/<snapshot_id>/manifest.json Corpus.graph_snapshot_manifest_path(extractor_id, snapshot_id)

The accessors compute paths from identifiers and do not check that the files exist. An extracted text artifact exists only when the snapshot produced text for that item, and a graph manifest exists only after the graph snapshot was built. extracted_text_path addresses pipeline extraction snapshots, whose extractor identifier is pipeline.

from biblicus.corpus import Corpus

corpus = Corpus.open("corpora/example")
catalog_path = corpus.catalog_path
text_path = corpus.extracted_text_path(extraction_snapshot_id, item_id)
manifest_path = corpus.graph_snapshot_manifest_path("cooccurrence", graph_snapshot_id)

The behavior is specified in features/corpus_layout_paths.feature.

Core vocabulary for this document

Day to day corpus workflows

These are the workflows that tend to matter most for a person building a corpus.

Decision points with options and recommendations

Each decision point includes a few viable options. These are not mutually exclusive in an absolute sense, but the project policy is one official way, so each section ends with a recommendation.

Locked decisions

The decisions below are treated as project policy for the hook system, and they are implemented in version zero.

Decision 1: corpus ignore rules

Goal: prevent accidental ingestion of irrelevant files such as build artifacts, large caches, and hidden metadata files.

Option A: a corpus ignore file stored in the corpus root

Option B: explicit ignore patterns passed to each command

Option C: a strict allow list instead of ignore patterns

Recommendation

Pick option A. A corpus should be self describing, and ignore rules are part of the corpus identity. A single ignore file is easy to document, easy to version, and easy to apply in every workflow.

Outcome

Version zero implemented option A with a .biblicusignore file in the corpus root.

Decision 2: large item ingestion and streaming

Goal: support large files and downloads without loading everything into memory.

Option A: keep the current bytes based ingestion only

Option B: add a streaming ingestion path that writes to disk in chunks

Option C: provide a dedicated download and ingest command that streams

Recommendation

Pick option B first. The core domain object is the item, and a stream is a natural way to represent large raw bytes. A dedicated download command can be layered on later as a convenience.

Outcome

Version zero implemented option B by adding a streaming ingestion path for local, non markdown files, with checksum recording.

Decision 3: content aware filename and media type detection

Goal: make raw files easy to open and inspect in a file manager, without guessing what the content is.

Option A: trust the response header and the address path

Option B: use light content sniffing for a small set of common formats

Option C: require explicit media type and filename in ingest calls

Recommendation

Pick option B. The benefit is daily usability. A user should be able to open a raw file and have the operating system recognize it. Light sniffing can be limited to a few signatures and still deliver most of the value.

Outcome

Version zero implemented option B for a small set of common file types when a web response is generic.

Decision 4: folder tree import semantics

Goal: import an existing library of files while preserving provenance.

Option A: preserve relative paths in-place under the corpus root

Option B: copy into a dedicated imported namespace

Option C: store references outside the corpus root

Recommendation

Pick option A. The corpus should be a stable source of truth without duplicating bytes. Files must live under the corpus root.

Outcome

Version zero implements option A. import-tree registers existing files in place under the corpus root.

Decision 5: website crawl scope and safety

Goal: ingest a small documentation site or a set of pages under a base address, without turning the tool into a general web crawler.

Option A: a strict base address prefix rule

Option B: a host based rule

Option C: no crawler, only single page ingestion

Recommendation

Pick option A. It is simple to explain, easy to test, and safe by default.

Outcome

Version zero locked this as policy. The crawler enforces the allowed prefix to keep scope explicit.

Decision 6: editorial workflow and reversible pruning

Goal: remove or hide items from active use without destroying the raw source material.

Option A: a soft delete flag in metadata

Option B: a multi layer corpus concept

Option C: move pruned items to an archive folder

Recommendation

Pick option A first. It aligns with the catalog model and keeps the raw bytes stable. Option C can be added later as a more explicit physical archive if users want it.

Outcome

Version zero locked this as policy. The prune workflow remains a future addition.

Decision 6A: derived artifact storage is partitioned by plugin type

Goal: retain derived artifacts from multiple implementations side by side so a user can compare results and switch between implementations without losing work.

This decision applies to extraction plugins and retrieval backends, and to any plugin type that produces derived artifacts.

Option A: store artifacts under the corpus, partitioned by plugin type

Suggested layout

Option B: store artifacts under the corpus but only partition by snapshot identifier

Option C: store artifacts outside the corpus in a workspace cache

Recommendation

Pick option A. It supports the comparison workflow directly and it makes the corpus folder a complete, portable unit for experimentation. The corpus root remains the source of truth, and derived artifacts remain clearly separated and rebuildable.

Outcome

This is implemented. Extraction, retrieval, analysis, and evaluation runs are stored under the corpus, partitioned by plugin type and identifier so multiple implementations can coexist.

Decision 6B: extraction is a separate plugin stage from retrieval

Goal: support experimentation where an extraction implementation and a retrieval implementation can be swapped independently while using the same corpus.

This decision was driven by optical character recognition, but it applies to any extraction method that converts source items into derived text artifacts.

Option A: extraction is embedded in the retrieval backend

Option B: extraction is embedded in corpus ingestion

Option C: extraction is a distinct plugin type and a distinct pipeline stage

Recommendation

Pick option C. It keeps the corpus raw and stable while allowing clean evaluation across extraction providers and across retrieval providers.

Outcome

This is implemented. Extraction is a distinct plugin stage with a command line interface entry point, and retrieval backends can reference a selected extraction snapshot.

Lifecycle hooks and where plugins can attach

The system already has a clear separation between raw items, the catalog, backend run builds, and retrieval queries. Lifecycle hooks make those boundaries explicit and give plugins a place to participate.

Hook points to consider

Ingestion

Catalog rebuild

Backend run build

Query and evidence

Evaluation

Decision 7: hook protocol design

Goal: allow plugins to attach to lifecycle points without turning the core into a plugin framework.

Option A: Python protocols with explicit hook interfaces

Option B: a generic event bus with untyped dictionaries

Option C: treat every hook as a command line tool

Recommendation

Pick option A. This project values explicit vocabulary and validation. A typed hook interface with Pydantic models makes behavior precise and keeps plugins honest.

Outcome

Version zero implemented option A with a small hook protocol, hook configuration validation, and built in example hooks.

Decision 8: how hook execution is recorded

Goal: a user should be able to answer what changed and why, especially when a plugin modifies metadata or produces derived artifacts.

Option A: a hook log file per run

Option B: write changes only into metadata

Option C: store a full audit trail database

Recommendation

Pick option A. A structured log file is readable, portable, and sufficient for early auditing.

Outcome

Version zero implemented option A by writing structured log entries for hook execution.

Outcomes and remaining questions

The hook protocol and hook logging policy above were implemented in version zero. This section records what was implemented and the open questions tracked for later iterations.

Hook contexts implemented in version zero

Hook log schema implemented in version zero

Remaining design questions

First behavior driven development slices implemented in version zero

These were small, concrete slices that were specified and built without committing to too much machinery.

Extension metadata

Downstream tools may attach their own data to a catalog item. That data lives under the item metadata mapping, in a key owned by the tool. Papyrus stores its extension fields under metadata.papyrus.

Migrating a catalog that has unknown top-level item fields:

  1. Preview the change with python scripts/migrate_catalog_extensions.py path/to/corpus --dry-run. Nothing is written.
  2. Run python scripts/migrate_catalog_extensions.py path/to/corpus. Unknown top-level item fields move under metadata. When the same key already exists under metadata and both values are mappings, they are merged and the top-level value wins on conflicts.
  3. The catalog is written atomically, and running the script again on a clean catalog changes nothing.

The behavior is specified in features/catalog_extension_metadata.feature.

Reproducibility checklist

Common pitfalls