Welcome to BenoStreamDB¶
BenoStreamDB is an index-overlay engine for the lakehouse built in Rust. It layers reconstructible, persistent secondary indexes—HNSW vector search, BM25 Okapi full-text, Roaring Bitmaps, JSON path filters, and CSR graphs—directly onto Apache Iceberg tables in object storage.
Tip
Looking for the fastest way to get started? Jump straight to the 2-Minute Quickstart to run your first vector search and SQL query in under 2 minutes.
Key Architecture Highlights¶
Serverless by Default: Run directly against object storage via an embedded library (Python, Rust, JVM) with zero database server overhead.
Advisory Index Overlays: Build rebuildable secondary indexes alongside standard Parquet data files. The underlying data is never rewritten or locked into a proprietary format.
Vector Search & pgvector SQL: Query vector indexes directly using standard pgvector SQL operators (
<->,<=>,<#>) in DataFusion.Graph Analytics & Graph RAG: Query CSR graph index overlays using native SQL table functions directly over Iceberg edge tables.
Ecosystem Connectors: Native connectors for Apache Spark, Trino, and dbt, plus optional Arrow Flight SQL and REST search gateways.
Grounded Hardware Support: Dynamic batch GPU acceleration via NVIDIA CUDA (runtime NVRTC), Apple Metal, and Vulkan/WGPU, with SIMD-optimized CPU execution as the fast, low-latency default.
Getting Started
- 2-Minute Quickstart
- ⚡ 1. Install (15 seconds)
- 🐍 2. Embedded Python: Indexes & Vector Search (45 seconds)
- 🔀 3. Hybrid Search: BM25 + Vector Fusion (20 seconds)
- 📊 4. Query with SQL & pgvector Syntax (30 seconds)
- 🏎️ 5. 4x Index Compression with TurboQuant™ (15 seconds)
- 🔌 Core Connectors: Flight SQL, Trino, Spark & dbt
- 🧭 Next Steps
- Installation
- 🔄 Fluent Query API
- Python Fluent API
- Rust Fluent API
- Benefits
- Python Vector Distance API with GPU Acceleration
- 📊 Real-World Testing Plan
- Phase 2: Nessie Integration (Next)
- Phase 3: Production Hardening ✅ COMPLETE
- 🏗️ Architecture
- 🔌 Connectors
- 🔨 Building Connectors
- 🧪 Development
- 📈 Roadmap
- 🤝 Contributing
- 📄 License
- 🙏 Acknowledgments
- Quickstart
- 1. The
bsdb-searchREST server - 2. The Python client
- Next steps
- Architecture: The Indexed Lakehouse
Architecture Decisions
- ADR-001: Tightened Overlay Index Artifact Contract & Compound Puffin Bundles
- 1. Executive Summary
- 2. Problem Statement & Motivation
- 3. The Tightened Index Artifact Specification
- 4. Compound Index Bundles (Apache Iceberg Puffin Specification)
- 5. Detached Overlay Catalogs for External Iceberg Tables
- 6. Merge-On-Read (MOR) Lifecycle for Secondary Indexes
- 7. Migration & Compatibility Plan
- 8. Alternatives Considered
- 9. Observability & Telemetry
- 10. Security Considerations
- Appendix A: Implementation Cross-Reference Map
- ADR-002: Global Vector Clustering via Index-Only Compaction
- ADR-003: Graph RAG Hybrid Memory (In-RAM State, Out-of-Core Edges)
- ADR-004: Observability Pipeline (Registry, Exporters, OTLP)
- ADR-005: SQL DDL and Maintenance Surface over the Core
TableAPI - ADR-006: Trino Vector and Graph Pushdown via Table Functions
- 1. Decision
- 2. What the code does today
- 3. Target SQL surface
- 4. Architecture
- 5. Phase 0: truth-in-docs (0.5 day)
- 6. Phase 1: index DDL (2–3 days)
- 7. Phase 2: vector search table function (4–6 days)
- 8. Phase 3: graph table functions (3–5 days)
- 9. Phase 4: hybrid RRF (8–12 days)
- 10. File-level change list
- 11. Test matrix
- 12. Risks and open questions
- 13. Estimates
- 14. Article implications
- ADR-007: Native Connector Architecture, UDF Exposure, and Version Matrix
- ADR-008:
PendingCommitTransaction State Machine, Linearizable Reads, and Segmented WAL Recovery - ADR-009: Catalog Parity, Compare-and-Swap Hardening, and the Cross-Catalog Conformance Contract
- ADR-010: Iceberg V3 Writes Must Be the Spec, and a Foreign Reader Is the Acceptance Test
Connectors & Integrations
Table Formats & Catalogs
- Iceberg Compatibility Matrix
- Iceberg V2/V3 API Documentation
- Multi-Catalog Usage Guide
- Supported Catalogs
- 0. Memory and JDBC Catalogs (Zero-Infrastructure & SQL Backends)
- 1. Hive Metastore (Detailed Example)
- 2. Project Nessie
- 3. AWS Glue Catalog & Amazon S3 Tables
- 4. Iceberg REST Catalog (Snowflake / Apache Polaris & Lakekeeper)
- Next Steps
- Snowflake Polaris
- Architecture Overview
- Prerequisites
- Step 1: Configure BenoStreamDB to Sync with Polaris
- Step 2: Configure Storage Access in Snowflake (External Volume)
- Step 3: Configure Polaris Catalog Integration in Snowflake
- Step 4: Create the Table in Snowflake
- Step 5: Ingest via BenoStreamDB and Query in Snowflake
- Key Benefits
Query & Search
- pgvector SQL Support Guide
- Python Vector Distance API Reference
- Overview
- Supported Distance Metrics
- GPU Backend Support
- Installation
- Quick Start
- API Reference
- Performance Optimization
- Integration with SQL
- Troubleshooting
- Examples
- See Also
- API reference
- Python API
- Spark Connector
- Trino Connector
- Configuration
- REST Search APIs (
benostreamdb-search) - Error Handling
- Performance Tips
- See Also
- Graph RAG & Iceberg Edge Tables Guide
- 1. Standard Edge Table Schema Convention
- 2. Automated Sidecar Indexing
- 3. NetworkX Interoperability
- 4. End-to-End Hybrid Graph RAG Workflow
- 5. Iceberg Partitioning & Optimization Best Practices
- 6. End-to-End Graph RAG Search & Community Summarization
- 7. Analytics Engineering with dbt (
dbt-benostreamdb)
- GPU Setup Guide for BenoStreamDB
Operations & Hardening
Tutorials
Benchmarks & Evaluation
- 1. The reproducibility contract
- 2. Prerequisites
- 3. Quick start
- 3.1 Running the whole sweep
- 4. Workload families
- 5. Resource envelope
- 6. Output schema
- 7. Correctness oracles
- 8. Regenerating the rollup and report
- 9. How to read the numbers
- 10. Troubleshooting
- BenoStreamDB Benchmarking Plan
- 1. Goals and non-goals
- 2. Datasets (external, standard, citable)
- 3. Competitor set
- 4. Harnesses
- 5. Resource-envelope rules
- 6. Metrics and output schema
- 7. Production workload classes
- 8. Methodology and reproducibility
- 9. Reporting format
- 10. Automation and CI
- 11. Known findings and current status
- 12. Roadmap
- BenoStreamDB — Release Readiness Assessment
API Reference