Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Chunking

Embedding retrieval depends on chunking.

Most corpora contain documents that are too long to retrieve effectively as single “whole-document” units. Biblicus therefore treats chunking as part of the retrieval indexing contract: embeddings are computed over chunks, and retrieval returns evidence with chunk boundaries so you can trace results back to the original item text.

Chunkers are pluggable

Chunking is a pluggable interface selected by identifier in a retrieval configuration:

There are no hidden fallbacks. If you select a chunker without providing required configuration, Biblicus fails with a user-facing error that explains what is missing.

Chunk identifiers and provenance

Chunks must be addressable and reproducible:

Evidence produced by embedding retrieval references chunk provenance so downstream tooling can reconstruct context packs without guessing.

Built-in chunking strategies

Biblicus provides multiple built-in chunking strategies so you can compare tradeoffs explicitly:

Fixed character window

Split text into windows of a fixed character length with optional overlap.

Typical parameters:

Paragraph chunking

Split text into paragraphs (blank-line delimited), with optional joining/splitting rules to avoid chunks that are too small or too large.

Typical parameters:

Fixed token window

Split text into windows of a fixed token length with optional overlap.

Token-based chunking depends on a tokenizer interface (see next section) so Biblicus can support multiple tokenization strategies without locking into a single library.

Tokenization is pluggable too

Token-based chunkers depend on a second pluggable interface selected by identifier:

This keeps the surface configurable while avoiding implicit dependencies. If a token-based chunker is selected without a tokenizer implementation configured, Biblicus fails fast with explicit guidance.