Biblicus

Make your documents usable by your assistant, then decide later how you will search and retrieve them.

Real-Time Subscriptions in Biblicus Dashboard

Overview

Amplify Gen 2 provides automatic real-time updates via AppSync subscriptions. When the Python CLI creates or updates DynamoDB records, the dashboard receives WebSocket events and re-renders automatically—no polling required.

This document explains how to use real-time subscriptions in Biblicus, with examples for both backend (Python) and frontend (React) code.

Architecture

Python CLI (AmplifyPublisher)
    ↓ HTTP GraphQL Mutation
AppSync API
    ↓ DynamoDB Stream
Real-Time Resolver
    ↓ WebSocket (graphql-ws)
Browser (observeQuery subscription)
    ↓ React State Update
Dashboard UI Re-renders

Key Components:

Pattern: State Changes via Table Records

Key Principle: State changes happen via record mutations, not arbitrary signals.

Instead of sending custom "snapshot started" or "progress update" messages, we create/update database records. AppSync automatically notifies subscribers of these changes via onCreate, onUpdate, and onDelete events.

Example: Extraction Progress

  1. CLI creates Snapshot with status='RUNNING', completedItems=0

    • Triggers onCreate subscription event
    • Dashboard sees new snapshot appear
  2. Dashboard subscribes to Snapshot changes via observeQuery()

    • Automatically receives all future updates
  3. CLI updates completedItems incrementally (10, 20, 30...)

    • Each update triggers onUpdate subscription event
    • Dashboard progress bar animates automatically
  4. CLI updates status='COMPLETED'

    • Triggers final onUpdate event
    • Dashboard shows completion indicator

This pattern comes "for free" from Amplify Gen 2—no custom WebSocket infrastructure needed.

Implementation Examples

Backend (Python)

from biblicus.sync.amplify_publisher import AmplifyPublisher

publisher = AmplifyPublisher('MyCorpus')

# Create snapshot (triggers onCreate subscription)
publisher.start_snapshot(
    snapshot_id='extraction-abc123',
    snapshot_type='EXTRACTION',
    total_items=100
)

# Update progress (triggers onUpdate subscription)
for i in range(0, 101, 10):
    publisher.update_snapshot_progress(
        snapshot_id='extraction-abc123',
        completed_items=i
    )
    time.sleep(1)

# Complete (triggers onUpdate subscription)
publisher.complete_snapshot(
    snapshot_id='extraction-abc123',
    total_items=100
)

How it works:

Frontend (React)

import { useEffect, useState } from 'react';
import { client } from '../lib/amplify-client';
import type { Snapshot } from '../lib/amplify-client';

function CorpusDashboardPage() {
  const [snapshots, setSnapshots] = useState<Snapshot[]>([]);

  useEffect(() => {
    // observeQuery returns subscription that receives:
    // - onCreate: new snapshot appears
    // - onUpdate: progress/status changes
    // - onDelete: snapshot removed
    const subscription = client.models.Snapshot.observeQuery({
      filter: { corpusId: { eq: corpusName } }
    }).subscribe({
      next: ({ items }) => {
        setSnapshots(items); // Automatic re-render
      },
      error: (err) => {
        console.error('Subscription error:', err);
      }
    });

    return () => subscription.unsubscribe();
  }, [corpusName]);

  return (
    <div>
      {snapshots.map(snapshot => (
        <ProgressBar
          key={snapshot.snapshotId}
          current={snapshot.completedItems}
          total={snapshot.totalItems}
          status={snapshot.status}
        />
      ))}
    </div>
  );
}

How it works:

Subscription Patterns

1. List All Items (with Real-Time Updates)

Subscribe to all catalog items for a corpus:

client.models.CatalogItem.observeQuery({
  filter: { corpusId: { eq: corpusName } }
}).subscribe({
  next: ({ items }) => setItems(items)
});

Use case: Catalog browser that shows new items as they're ingested

2. Filtered Query (with Real-Time Updates)

Subscribe only to items matching specific criteria:

client.models.FileMetadata.observeQuery({
  filter: {
    corpusId: { eq: corpusName },
    status: { eq: 'UPLOADING' }
  }
}).subscribe({
  next: ({ items }) => setUploadingFiles(items)
});

Use case: Upload progress tracker showing only in-progress uploads

3. Single Item Watch

Subscribe to updates for a specific snapshot:

client.models.Snapshot.observeQuery({
  filter: {
    corpusId: { eq: corpusName },
    snapshotId: { eq: snapshotId }
  }
}).subscribe({
  next: ({ items }) => setSnapshot(items[0])
});

Use case: Snapshot detail page showing real-time progress

4. Secondary Index Query

Use GSI for efficient filtered subscriptions:

// Requires GSI defined in schema
client.models.CatalogItem.itemsByMediaType({
  mediaType: 'application/pdf',
  corpusId: corpusName
}).subscribe({
  next: ({ items }) => setPdfItems(items)
});

Use case: Media-type filtered catalog view

Error Handling

AppSync subscriptions automatically reconnect on network failure. Handle transient errors gracefully:

const subscription = client.models.Snapshot.observeQuery({ filter })
  .subscribe({
    next: ({ items }) => {
      setSnapshots(items);
      setError(null); // Clear error on successful update
    },
    error: (err) => {
      console.error('Subscription error:', err);
      setError('Connection lost. Retrying...');
      // AppSync will automatically reconnect
      // Show user-friendly message while reconnecting
    }
  });

Best practices:

Performance Considerations

Subscription Limits

Message Size

Bandwidth

Battery Impact

Optimization Tips

  1. Minimize subscriptions: Combine filters when possible
  2. Use GSIs: Avoid client-side filtering of large result sets
  3. Pagination: Use limit and nextToken for large catalogs
  4. Conditional rendering: Only render visible items

Testing

Manual Test (CLI Publisher)

Test real-time updates by simulating an extraction:

# Terminal 1: Watch dashboard
cd apps/dashboard && npm run dev
open http://localhost:5175

# Terminal 2: Simulate extraction
python scripts/test_snapshot_realtime.py

Expected behavior:

  1. New snapshot appears within 1 second
  2. Progress bar animates from 0→100%
  3. Status changes to COMPLETED
  4. All updates smooth and immediate

Automated Test (E2E)

Run end-to-end subscription tests:

python scripts/test_realtime_e2e.py

Validates:

Troubleshooting

Subscription Not Receiving Updates

Symptoms: Dashboard doesn't update when CLI makes changes

Diagnosis:

  1. Check Amplify configuration: apps/dashboard/amplify_outputs.json
    cat apps/dashboard/amplify_outputs.json | jq '.data.url'
    
  2. Verify API key hasn't expired (check Amplify console)
  3. Inspect browser network tab for WebSocket connection
    • Look for wss:// connection to AppSync
    • Should show as "101 Switching Protocols"
  4. Check console for @connection errors

Solutions:

Updates Delayed

Symptoms: Changes take 2-5 seconds to appear in dashboard

Normal latency: 100-500ms (AppSync → Browser)

High latency (>2s):

Solutions:

Memory Leaks

Symptoms: Browser tab memory grows over time, eventually slows down

Cause: Forgot to unsubscribe in useEffect cleanup

Solution: Always unsubscribe in cleanup function:

useEffect(() => {
  const sub = client.models.Snapshot.observeQuery(...).subscribe(...);
  return () => sub.unsubscribe(); // Critical!
}, [dependencies]);

Validation:

Authentication Errors

Symptoms: "Not Authorized to access..." errors in console

Cause: Wrong authorization mode or expired credentials

Solutions:

WebSocket Disconnections

Symptoms: Subscription stops working after 5-10 minutes

Cause: Network timeout, browser throttling, or AppSync connection limit

Normal behavior: AppSync automatically reconnects

Monitor reconnections:

let reconnectCount = 0;

const subscription = client.models.Snapshot.observeQuery({ filter })
  .subscribe({
    next: ({ items }) => {
      console.log(`Received update (reconnect count: ${reconnectCount})`);
      setSnapshots(items);
    },
    error: (err) => {
      reconnectCount++;
      console.error(`Disconnected (attempt ${reconnectCount}):`, err);
    }
  });

When to worry:

Advanced Patterns

Optimistic Updates

Show changes immediately, then reconcile with server:

function addItem(item: CatalogItem) {
  // Show immediately
  setItems(prev => [...prev, item]);

  // Send to server
  client.models.CatalogItem.create({ input: item }).then(() => {
    // Subscription will reconcile actual server state
  }).catch(err => {
    // Revert optimistic update on error
    setItems(prev => prev.filter(i => i.itemId !== item.itemId));
  });
}

Pagination with Real-Time

Combine pagination with subscriptions for large catalogs:

const [items, setItems] = useState<CatalogItem[]>([]);
const [nextToken, setNextToken] = useState<string | null>(null);

// Initial load with pagination
useEffect(() => {
  client.models.CatalogItem.list({
    filter: { corpusId: { eq: corpusName } },
    limit: 50,
    nextToken: nextToken
  }).then(({ data, nextToken: next }) => {
    setItems(prev => [...prev, ...data]);
    setNextToken(next);
  });
}, [corpusName, nextToken]);

// Real-time updates for new items
useEffect(() => {
  const subscription = client.models.CatalogItem.observeQuery({
    filter: {
      corpusId: { eq: corpusName },
      createdAt: { gt: new Date().toISOString() } // Only new items
    }
  }).subscribe({
    next: ({ items: newItems }) => {
      setItems(prev => [...newItems, ...prev]); // Prepend new items
    }
  });

  return () => subscription.unsubscribe();
}, [corpusName]);

Conditional Subscriptions

Enable/disable subscriptions based on user preference:

function useConditionalSubscription(enabled: boolean) {
  const [items, setItems] = useState<Snapshot[]>([]);

  useEffect(() => {
    if (!enabled) {
      // Poll instead
      const interval = setInterval(() => {
        client.models.Snapshot.list({ filter }).then(({ data }) => {
          setItems(data);
        });
      }, 5000);

      return () => clearInterval(interval);
    }

    // Subscribe for real-time
    const sub = client.models.Snapshot.observeQuery({ filter })
      .subscribe({ next: ({ items }) => setItems(items) });

    return () => sub.unsubscribe();
  }, [enabled]);

  return items;
}

Summary

Key Takeaways:

  1. State changes via DynamoDB records trigger automatic WebSocket updates
  2. Backend uses HTTP mutations only (no WebSocket code)
  3. Frontend uses observeQuery() for automatic subscriptions
  4. AppSync handles reconnection, batching, and scaling automatically
  5. Pattern comes "for free" from Amplify Gen 2

When to use subscriptions:

When to use polling: