Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Metadata Text Extractor

Extractor ID: metadata-text

Category: Text/Document Extractors

Overview

The metadata text extractor generates searchable text representations from catalog item metadata. Instead of processing file content, it creates small, stable text artifacts from titles and tags stored in the corpus catalog.

This extractor is designed for retrieval over non-text items like images, audio, or binary files where the metadata provides the primary semantic signal. It's also useful for comparing retrieval backends while keeping extraction deterministic and stable. For topic modeling, explicit field selection can emit title and abstract text without leaking tags or curation metadata into discovery.

Installation

No additional dependencies required. This extractor is part of the core Biblicus installation.

pip install biblicus

Supported Media Types

All media types are supported. The extractor processes any catalog item that has metadata (title or tags).

Configuration

Config Schema

class MetadataTextExtractorConfig(BaseModel):
    include_title: bool = True   # Include item title
    include_tags: bool = True    # Include tags line
    fields: list[str] = []       # Explicit field paths to emit

Configuration Options

Option Type Default Description
include_title bool true Include the item title as the first line
include_tags bool true Include a tags: ... line if tags exist
fields list[str] [] Explicit field paths to emit instead of the default title/tags behavior

When fields is present, only those fields are emitted. Supported paths are title, tags, and metadata.<key>.

Usage

Command Line

Basic Usage

# Extract metadata as text
biblicus extract my-corpus --extractor metadata-text

Custom Configuration

# Only include titles, skip tags
biblicus extract my-corpus --extractor metadata-text \
  --config include_tags=false

Configuration File

extractor_id: metadata-text
config:
  include_title: true
  include_tags: true

For blind topic discovery over research papers, use explicit metadata fields:

extractor_id: metadata-text
config:
  fields:
    - title
    - metadata.abstract

This excludes tags, curation.*, and other metadata by construction.

biblicus extract my-corpus --configuration configuration.yml

Python API

from biblicus import Corpus

# Load corpus
corpus = Corpus.from_directory("my-corpus")

# Extract with defaults
results = corpus.extract_text(extractor_id="metadata-text")

# Extract with custom config
results = corpus.extract_text(
    extractor_id="metadata-text",
    config={
        "include_title": True,
        "include_tags": False
    }
)

In Pipeline

Metadata Fallback

extractor_id: pipeline
config:
  stages:
    - extractor_id: pass-through-text
    - extractor_id: metadata-text  # Fallback for non-text items
    - extractor_id: select-text

Image Metadata Retrieval

extractor_id: select-smart-override
config:
  default_extractor: metadata-text
  overrides:
    - media_type_pattern: "image/.*"
      extractor: metadata-text

Examples

Basic Metadata Extraction

Given a catalog item with metadata:

id: photo-001
media_type: image/jpeg
title: "Sunset over mountains"
tags: ["nature", "landscape", "golden-hour"]

Extracted text:

Sunset over mountains
tags: nature, landscape, golden-hour

Title-Only Extraction

Extract just titles for minimal overhead:

biblicus extract photos-corpus --extractor metadata-text \
  --config include_tags=false

Output:

Sunset over mountains

Image Corpus Retrieval

Create searchable text for an image collection:

from biblicus import Corpus

# Corpus of photos with descriptive metadata
corpus = Corpus.from_directory("photos")

# Extract metadata as text for retrieval
results = corpus.extract_text(extractor_id="metadata-text")

Mixed Media Pipeline

Use metadata for items that can't be processed:

extractor_id: pipeline
config:
  stages:
    - extractor_id: pass-through-text
    - extractor_id: ocr-rapidocr
    - extractor_id: metadata-text    # Catch-all for remaining items
    - extractor_id: select-text

Output Format

The output is plain text formatted as:

  1. Title line (if include_title is true and title exists): The item title as-is
  2. Tags line (if include_tags is true and tags exist): tags: tag1, tag2, tag3

Both elements are optional. If neither title nor tags exist, the extractor returns None.

Behavior Details

Empty Metadata

Items without title or tags (or with both disabled) return None, causing the extractor to skip them.

Whitespace Handling

Tag Formatting

Tags are joined with commas and spaces: tags: tag1, tag2, tag3

Deterministic Output

Output is completely deterministic based on catalog metadata. No file I/O is performed, making this extractor extremely stable for benchmarking retrieval systems.

Performance

This is one of the fastest extractors as it only accesses in-memory catalog metadata.

Error Handling

Missing Metadata

Items without applicable metadata are silently skipped (returns None).

Invalid Metadata Types

Non-string titles or tags are filtered out. The extractor is defensive against malformed catalog data.

Use Cases

Image Retrieval

Build text indices for image collections:

biblicus extract photos --extractor metadata-text

Create searchable text for music or podcast libraries:

biblicus extract music-library --extractor metadata-text

Retrieval Benchmarking

Compare retrieval backends with stable extraction:

from biblicus import Corpus

corpus = Corpus.from_directory("benchmark-corpus")

# Stable extraction for fair comparison
results = corpus.extract_text(extractor_id="metadata-text")

Non-Text Fallback

Provide searchable text for items that can't be processed:

extractor_id: pipeline
config:
  stages:
    - extractor_id: docling-smol
    - extractor_id: metadata-text  # Fallback
    - extractor_id: select-text

Best Practices

When to Use Metadata Extractor

Use metadata-text when:

Don't use metadata-text when:

Metadata Quality

The effectiveness of this extractor depends entirely on metadata quality:

Catalog Preparation

Ensure your catalog has good metadata:

# Good metadata
id: research-paper-001
title: "Neural Networks for Document Understanding"
tags: ["ml", "nlp", "research", "deep-learning"]

# Poor metadata
id: doc001
title: "Document 1"
tags: []

Same Category

Alternatives

Pipeline Utilities

See Also