MuSiQue Multi-Hop GraphRAG Benchmark & Parity Evaluation¶

This document details the official results and methodology from the MuSiQue multi-hop GraphRAG benchmark:

  1. Engine-Native Passage-Level Retrieval (Headline): BenoStreamDB’s in-engine retrieval, with no hosted LLM in the loop, clears Zilliz’s published Naive RAG baseline (0.5943 vs. 0.5803 Micro Recall@5).

  2. Entity-Level Vector Graph RAG (Architecture Demo): Demonstrates BenoStreamDB natively serving the three-collection Passages/Entities/Relations/Mentions layout directly over Apache Iceberg tables without requiring external vector or graph databases.

Both benchmarks utilize the exact upstream inputs published by Zilliz and HippoRAG, ensuring a like-for-like empirical comparison on the corpus, the knowledge graph, and the 1,000-question evaluation manifest.

1. Hardware & Resource Envelope¶

Hardware & Memory Distinction (MuSiQue vs. NeurIPS BigANN):

  • MuSiQue GraphRAG: Constrained to 8 vCPUs / 32 GiB RAM (--memory=32g in benchmarks/docker-compose.musique.yml and benchmarks/run_musique_docker.sh). The 32 GiB memory budget is strictly required because the runner hosts the local in-memory PyTorch runtime for bge-large-en-v1.5 dense embeddings and the BGE cross-encoder reranker, alongside four active Iceberg tables.

  • NeurIPS BigANN: Constrained to 8 vCPUs / 16 GiB RAM (Azure Standard_D8lds_v5). BigANN exercises raw columnar Parquet streaming and HNSW/IVF vector indexing without hosted transformer models.

Parameter

MuSiQue GraphRAG Specification

Reference Envelope

8 vCPUs, 32 GiB RAM (deploy.resources.limits.memory: 32G)

Embedding Model

BAAI/bge-large-en-v1.5 (1,024 dimensions, Float32)

Reranker Model

BAAI/bge-reranker-large (Cross-encoder)

Storage Tables

Native Apache Iceberg (V2/V3) Parquet

Index Overlays

HNSW (cosine metric) + CSR Graph Indexes

Host Execution

Linux x86_64 / NVIDIA CUDA or Apple Silicon Metal

2. Engine-Native Passage-Level Retrieval (Headline)¶

Evaluated on the official Zilliz manifest sample of 1,000 multi-hop questions using bge-large-en-v1.5 (1,024 dimensions), Float32 vectors, and BGE cross-encoder reranking. Zero hosted LLMs, zero outbound API calls, zero per-token cost.

Retrieval Strategy

Micro Recall@5

Description

Hop-1 Vector Search

0.5600

Direct top-$k$ dense vector search on passage embeddings.

Native Personalized PageRank (graph_rag_search)

0.5520

In-engine PPR traversal across passage co-occurrence graph.

Two-Stage + BGE Cross-Encoder

0.5943

Vector search + DataFusion graph expansion + local cross-encoder.

Comparison Baseline:
Zilliz’s published Naive RAG baseline (BGE-large-en-v1.5) is 0.5803.
BenoStreamDB’s two-stage pipeline scores 0.5943 Micro Recall@5 (+1.40 points) with the entire retrieval cycle executing in-process inside the database engine.

3. Entity-Level Vector Graph RAG (Architecture Demo)¶

The same 1,000-question manifest evaluated across four native Apache Iceberg tables rather than passage-level co-occurrence:

Table Name

Contents & Schema

Index Type

musique_entity_rag_passages

11,656 passages (id, doc_key, title, text, embedding)

HNSW vector embedding

musique_entity_rag_entities

64,982 deduplicated OpenIE entities (id, name, embedding)

HNSW vector embedding

musique_entity_rag_relations

108,594 entity↔entity edges (source, target, relation, weight)

CSR graph index

musique_entity_rag_mentions

108,709 entity→passage edges (source, target, relation, weight)

CSR graph index

Retrieval Traversal Pipeline¶

  1. Vector-search the Entities table for query entity seeds.

  2. Breadth-First Search (BFS) the Relations edge table to discover related entities.

  3. Follow Mentions edges from discovered entities to candidate passages.

  4. Cross-encoder rerank the candidate passages.

Strategy

Micro R@5

Hit@5

All-hit@5

Precision@5

Latency (p50)

vector (passage baseline sanity check)

0.5597

0.9310

0.2260

0.2774

1.34 ms

entity_seed (no relation expansion)

0.3228

0.6770

0.0520

0.3342

7.05 ms

entity_bfs (seed + relation expansion)

0.1348

0.2740

0.0270

0.0884

19.49 ms

entity_rag (expansion + BGE rerank)

0.2140

0.4140

0.0590

0.1265

103.10 ms

Systems & Algorithmic Findings¶

  1. Passage Baseline Verification: The vector row reproduces the passage baseline (0.5597 ≈ 0.5600), verifying the underlying Iceberg table build, Arrow FFI, and HNSW vector index paths.

  2. The LLM Upper-Band Trade-off: Naive OpenIE triple traversal without relationship pruning does not match HippoRAG/Zilliz’s ~73% upper band. Entity-name embeddings are short and noisy, and untyped edges cause hub-entity flooding. The upper band reported by HippoRAG requires an application-layer LLM relationship-selection filter—workflow logic that lives above the storage engine, not inside the database.

4. Upstream Data Sources¶

All inputs are identical to the published Zilliz and HippoRAG artifacts:

Artifact

Source Repository / Path

Volume

musique_corpus.json

zilliztech/vector-graph-rag

11,656 passages

hippo_openie.json

OSU-NLP-Group/HippoRAG

64,982 entities

zilliz_manifest.json

zilliztech/vector-graph-rag

1,000 questions

Download commands:

curl -L https://raw.githubusercontent.com/zilliztech/vector-graph-rag/main/evaluation/data/musique_corpus.json \
  -o benchmarks/musique_corpus.json

curl -L https://raw.githubusercontent.com/OSU-NLP-Group/HippoRAG/5ddedf0f516c7bbed777ba54da680c8bb8fb8f84/output/openie_musique_results_ner_gpt-3.5-turbo-1106_11656.json \
  -o benchmarks/hippo_openie.json

curl -L https://raw.githubusercontent.com/zilliztech/vector-graph-rag/main/evaluation/jev/two-stage/manifest.json \
  -o benchmarks/zilliz_manifest.json

5. Reproduction Harness¶

1. Engine-Native Passage-Level Retrieval (Headline, 0.5943)¶

python3 benchmarks/run_musique_parity.py --mode f32 --reranker bge --rebuild \
  --corpus-json benchmarks/musique_corpus.json \
  --triplets-json benchmarks/hippo_openie.json \
  --sample-manifest benchmarks/zilliz_manifest.json

2. Entity-Level Architecture Demo¶

python3 benchmarks/run_musique_entity_rag.py --mode f32 --reranker bge --rebuild \
  --corpus-json benchmarks/musique_corpus.json \
  --triplets-json benchmarks/hippo_openie.json \
  --sample-manifest benchmarks/zilliz_manifest.json

3. Docker Containerized Reproduction (32 GiB Enforced)¶

# Build the benchmark image
docker build -f benchmarks/Dockerfile.musique -t bsdb-musique .

# Execute under strict cgroup limits (32G memory)
docker compose -f benchmarks/docker-compose.musique.yml run --rm musique