BenoStreamDB Recovery Runbook¶

This document outlines the formal recovery procedures for BenoStreamDB, detailing RPO/RTO expectations and step-by-step resolution paths for catastrophic failures.

BenoStreamDB is designed around a strict separation of authoritative state (Iceberg snapshots, Parquet data files) and derived state (HNSW/BM25 indexes). This architecture ensures that most “database corruption” events are merely cache-invalidation events that can be resolved without data loss.

Recovery Objectives¶

RPO (Recovery Point Objective)¶

BenoStreamDB’s RPO depends entirely on your configured WalDurability setting (BSDB_WAL_DURABILITY / TableBuilder::with_durability):

  • WalDurability::Sync ("sync", Default): RPO = 0. Every acknowledged write waits for fdatasync (File::sync_data, falling back to File::sync_all) on the WAL segment before the caller receives Ok. Failed syncs return an error to the caller. Note: for s3://, gs://, or az:// tables, configure a persistent local volume via BSDB_WAL_DIR or TableBuilder::with_wal_dir(); otherwise the WAL defaults to temporary local storage (/tmp/benostream_wal/...) and logs a startup warning.

  • WalDurability::Async ("async"): RPO <= sync window. Batches are handed to the WAL worker channel without blocking for fdatasync; the worker syncs every BSDB_WAL_SYNC_BATCH_SIZE (10 batches) or BSDB_WAL_SYNC_INTERVAL_MS (100 ms). Writes buffered inside the un-synced window are lost on hard termination.

RTO (Recovery Time Objective)¶

  • Data Availability: Seconds. On startup, BenoStreamDB reads the latest manifest (_manifest/v{N}.json), replays uncommitted records from the segmented WAL while skipping any benostream:tx_id already present in the manifest property benostream.committed_wal_tx, and immediately resumes queries (falling back to Parquet scans while secondary indexes rebuild).

  • Performance Recovery (Index Rebuild): Minutes to Hours. If an index is lost or corrupted, queries degrade deterministically to Parquet/exact scans. Rebuilding the index executes asynchronously without blocking ingestion.

1. Restore-from-Object-Store Procedure¶

Because Iceberg/Parquet in the object store is the authoritative source of truth, BenoStreamDB compute nodes are essentially stateless cache engines.

Scenario: The physical server hosting BenoStreamDB is destroyed. Action:

  1. Provision a new compute node.

  2. Ensure the node has IAM/network access to the backing object store (S3/GCS/Azure).

  3. Start the BenoStreamDB process pointing to the existing URI (with BSDB_WAL_DIR pointing to persistent local WAL storage if present):

    let table = Table::new_async("s3://my-bucket/my-table").await?;

  4. Result: The system loads the latest atomic manifest (_manifest/v{N}.json via load_latest_direct) and Iceberg metadata pointer, replays any uncommitted WAL segments, and resumes reads and writes.

2. WAL Recovery Procedure¶

The Write-Ahead Log (_wal/ by default for local tables, or BSDB_WAL_DIR for remote URIs) stores durable writes in rotated Arrow IPC stream segments named log_<timestamp>_<pid>_<seq>.arrow until the transaction commits to a manifest and watermark pruning removes the sealed file (prune_committed_segments).

Automatic Startup Recovery (Standard Crashes & Torn Tails)¶

In most crash scenarios (SIGKILL, power loss mid-append, or crash between manifest commit and WAL cleanup), no manual action is required:

  • Torn Trailing Frame / Interrupted Header: On startup, open_wal_segment and SafeIpcSegmentIter automatically recover from 0-byte segments, zeroed preallocation headers (<= 8 bytes), truncated Arrow IPC continuation/schema headers at EOF (Empty), and an UnexpectedEof mid-way through a trailing uncommitted batch frame (logging a warn and preserving every complete synced prefix batch).

  • Idempotent Crash Replay: If the process exited after committing _manifest/v{N}.json (CommitPhase::ManifestCommitted) but before watermark pruning deleted the sealed .arrow segment, startup filters out every batch whose benostream:tx_id already appears in benostream.committed_wal_tx and registers the segment for pruning on the next commit.

Manual Quarantine Procedure (Mid-Stream Byte Corruption)¶

Scenario: Storage hardware corruption damages bytes inside a WAL header (non-zero wrong magic bytes, invalid metadata length) or mid-stream FlatBuffers payload, causing startup to refuse initialization with a WAL Recovery Error.

Action: BenoStreamDB refuses silent data truncation when mid-stream corruption is detected inside a non-EOF record. To restore table availability by quarantining only the damaged segment:

  1. Inspect startup logs for the failing segment path (under [table_uri]/_wal/ or $BSDB_WAL_DIR).

  2. Inspect the directory (ls -la sorted chronologically; filenames encode nanosecond timestamps and sequence numbers). Keep all undamaged segments in place.

  3. Quarantine only the failing segment outside the WAL directory:

    mkdir -p /var/backups/bsdb-wal-quarantine
mv /path/to/table/_wal/log_0001762780000000000_004210_00000003.arrow /var/backups/bsdb-wal-quarantine/

  4. Restart the process.

  5. Result: Startup replays all valid WAL segments, skips already-committed transactions idempotently, and resumes serving traffic. Only uncommitted batches contained after the corruption point in the quarantined segment are absent.

3. Index Rebuild Procedure¶

Indexes (HNSW, BM25, Bloom filters) are non-authoritative. If they become corrupted, out-of-sync, or are accidentally deleted, they can be regenerated directly from the Parquet data files.

Scenario: Vector search latency spikes because the HNSW index was partially deleted, or logs report IndexChecksumMismatch. Action:

  1. Drop the corrupted index. This removes the metadata pointer and deletes the corrupted .bin files:

    table.drop_index("embedding").await?;

  2. Re-create the index. The database will launch a background task to rebuild it from the authoritative Iceberg data:

    table.add_index("embedding".to_string(), IndexAlgorithm::HNSW).await?;

  3. Result: The system serves vector similarity queries using exact nearest-neighbor scans until the background HNSW build attaches to the manifest, then switches queries onto the rebuilt HNSW index automatically.

4. Catalog Recovery Procedure¶

If you are using a decoupled catalog (e.g., Nessie, REST, or Hive Metastore) to track table states instead of the built-in file-based commit.lock:

Scenario: The external catalog database (e.g., PostgreSQL backing Nessie) is irrecoverably corrupted. Action: Because BenoStreamDB writes valid Iceberg metadata files (v1.metadata.json, v2.metadata.json) into the metadata/ directory of the object store, the table can be recovered even if the catalog is lost.

  1. Inspect the metadata/ directory in your object store.

  2. Identify the highest numbered metadata file (e.g., v42.metadata.json).

  3. Manually register this table in your new/restored catalog using the Iceberg REST API or catalog-specific CLI:

    POST /v1/namespaces/default/register HTTP/1.1
Content-Type: application/json

{
  "name": "my-table",
  "metadata-location": "s3://my-bucket/my-table/metadata/v42.metadata.json"
}

  4. Result: The new catalog will point to the exact transaction state of v42, and BenoStreamDB will resume operations normally.