Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Typed entities and relations

A publication has its own idea of what matters: makers, models, licences, tasks, datasets. spaCy labels such as ORG and PERSON do not say that. The typed-entity extractor reads the extracted text of corpus items and returns entities and relations whose kinds and types come from a closed vocabulary that you supply. Biblicus never hard-codes a kind.

Every entity and relation carries its evidence: the item identifier and character offsets into the item text. Nothing is published or accepted here. An entity is either resolved to something the desk already accepted, or proposed for the desk to review; deciding which proposals to accept belongs to the consumer.

The backend was chosen by an evaluation, see Typed extractor evaluation. No language model is involved.

Install

The default backend uses the GLiNER2 span model. Its dependencies (PyTorch and Transformers) are heavy, so they live behind an optional extra:

pip install "biblicus[typed-extraction]"

Without the extra, --backend registry-only still works and --backend gliner2 fails with an error that names the extra. The model, fastino/gliner2-base-v1 at revision f9634218e53580c56edf0de97ca1a7d3f1c2354e (Apache-2.0), downloads from Hugging Face on first use and then runs on the CPU. Run the command as a job, not inside a Lambda bundle: the model needs about 2 GB of memory.

Run it

Build an extraction snapshot first, because the extractor reads extracted text:

biblicus extract build --corpus corpora/shipped --stage pass-through-text
biblicus extract typed-entities \
  --corpus corpora/shipped \
  --vocabulary vocabulary.yml \
  --registry registry.json \
  --out out/
Option Meaning
--corpus Corpus to read. Defaults to the corpus found from the working directory.
--vocabulary Closed vocabulary file (required).
--registry Accepted entities. Without it every mention is proposed.
--backend gliner2 (default): registry matcher first, then GLiNER2. registry-only: registry matcher alone.
--threshold Score threshold for the model, greater than 0 and at most 1. Default 0.7.
--item-id Item to process. Repeat for several. Default: every item that has extracted text.
--extraction-snapshot extractor_id:snapshot_id supplying the text. Default: the latest extraction snapshot.
--out Directory that also receives output.json.

Standard output is the output document. The same document is always written to the corpus, so --out is only for handing it to another system.

The vocabulary

A YAML (or JSON) file with the entity kinds and relation types of one publication. label is the only instruction the model receives for a kind or type, so keep it short and natural. description is for human readers.

version: 1
publication: shipped
entity_kinds:
  - key: maker
    label: company or developer
  - key: model
    label: AI model
  - key: licence
    label: software licence
relation_types:
  - key: made_by
    label: made by
  - key: licensed_under
    label: licensed under

A vocabulary can hold both model (a specific named model, such as CosyVoice2 or CosyVoice 3) and model_family (the line of models above it, such as CosyVoice). They are separate kinds and the extractor never folds one into the other. To relate a model to its family, add a member_of relation type (see datasets/typed_extraction_lab/vocabulary.yml). The model proposes member_of only when the text states the membership; a shared name prefix does not make one, and a model whose family the text does not name has no member_of relation.

Keys and labels must be unique inside their group. Every kind and type in the output is a key from this file. A complete example is datasets/typed_extraction_lab/vocabulary.yml.

The registry

A JSON list of accepted entities. aliases are the surface forms that identify the entity:

[
  {"key": "accuknox-inc", "kind": "maker", "aliases": ["AccuKnox Inc.", "AccuKnox"]},
  {"key": "apache-2-0", "kind": "licence", "aliases": ["Apache 2.0"]}
]

The extractor rejects a registry whose kind is outside the vocabulary, whose alias is empty after normalisation, or whose alias belongs to two different entities. The key itself is not an alias; list it if it should match.

How a mention is resolved

  1. Registry matcher. Every alias is searched in the item text. Aliases compare equal after normalisation: letters are lower-cased and every run of characters that is not a letter or digit counts as one space, so NAME-A, name a, Name A and NAME_A are the same alias. A match must cover whole words. Gemma 3 does not match inside Gemma-3-4B-it, and Qwen3 does not match inside Qwen3.5-4B: a text that continues into a longer name is a different mention. The longest alias wins where aliases overlap.
  2. Model. The model proposes spans for the remaining text. A span that overlaps a registry match is dropped. A span whose whole normalised text equals an alias of the same kind also resolves; anything else is proposed.

Resolution is never fuzzy and never by prefix. A mention that matches no alias exactly is a proposed entity, however similar it looks to a known one. Merging two entities is a decision for the desk.

Relations

Relations come from the model only; the registry matcher does not extract them. Both endpoints must be entities of the same item: an endpoint is the entity whose span overlaps the model's endpoint by at least half, and a relation with an endpoint that is not an entity is dropped. A relation is resolved only when both endpoints are resolved; otherwise it is proposed. The evaluation found relations less reliable than entities (relation F1 56 against 80 for a language model), which is why a relation with a proposed endpoint stays a proposal.

Output

Standard output, and output.json, is one document: a TypedEntitiesOutput. The published schema is docs/api-schemas/extract-typed-entities.response.schema.json; the request schema is extract-typed-entities.request.schema.json.

{
  "backend": {"id": "gliner2", "model": "fastino/gliner2-base-v1", "revision": "f9634218e53580c56edf0de97ca1a7d3f1c2354e", "threshold": 0.7},
  "items": [
    {
      "entities": [
        {"entity_key": "accuknox inc", "id": "e1", "kind": "maker", "score": 1.0,
         "span": {"end": 12, "start": 0}, "state": "resolved", "text": "AccuKnox Inc"},
        {"entity_key": "guard 1 1", "id": "e2", "kind": "model", "score": 1.0,
         "span": {"end": 32, "start": 23}, "state": "resolved", "text": "Guard 1.1"}
      ],
      "item_id": "item-0h",
      "relations": [
        {"id": "r1", "object": "e1", "score": 0.9754, "span": {"end": 32, "start": 0},
         "state": "resolved", "subject": "e2", "type": "made_by"}
      ]
    }
  ],
  "schema_version": 1,
  "summary": {"entities": 2, "items": 1, "proposed": 0, "relations": 1, "resolved": 2},
  "vocabulary": {"hash": "...", "publication": "shipped", "version": 1}
}

(Keys are sorted; the example is shortened.) span offsets are character offsets into the extracted text of the item, entities are ordered by position, and entity_key is set exactly when state is resolved. A registry match scores 1.0. A relation span covers both endpoints. A trailing full stop is not part of a span, so the alias AccuKnox Inc. matches the text AccuKnox Inc. as the span AccuKnox Inc.

Determinism and the cache

The output has no timestamps: the same text, vocabulary, registry and settings always produce byte-identical output. What the model returns for a text is cached under

analysis/typed-entities/cache/<key>.json

where the key is the hash of the text hash, vocabulary hash, backend, model, model revision and threshold. A second run with the same inputs reads the cache, never loads the model, and prints identical output. The cache holds the model's raw proposals, so changing the registry changes the resolution without calling the model again, while changing the vocabulary, the threshold or the text calls it again.

Each run is stored under a run identifier derived from all of its inputs:

analysis/typed-entities/<run id>/output.json      the output document
analysis/typed-entities/<run id>/metadata.json    when it ran, elapsed seconds, backend calls, cache hits, file paths

metadata.json is where anything that varies between executions lives.

Python

from biblicus.api import TypedEntitiesRequest, extract_typed_entities

output = extract_typed_entities(
    TypedEntitiesRequest(
        corpus_path="corpora/shipped",
        vocabulary_path="vocabulary.yml",
        registry_path="registry.json",
    )
)

The extractor depends only on biblicus.typed_extraction_backends.TypedExtractionBackend. A different backend implements extract(text, vocabulary, threshold) and returns character offsets; it is passed to biblicus.typed_extraction.extract_typed_entities directly.

Try it on the lab dataset

datasets/typed_extraction_lab/ holds 142 public documents, a vocabulary and gold labels. prepare_corpus.py turns them into a corpus and builds a registry from half of the gold labels:

python scripts/typed_extraction_evaluation/prepare_corpus.py --out /tmp/typed-real
biblicus extract typed-entities --corpus /tmp/typed-real/corpus \
  --vocabulary datasets/typed_extraction_lab/vocabulary.yml \
  --registry /tmp/typed-real/registry.json --backend gliner2 --out /tmp/typed-real/out

The specifications are features/typed_entity_extraction.feature, which runs against a fake GLiNER2 library, and features/typed_entity_extraction_integration.feature, which runs the pinned model when it is already in the local Hugging Face cache.