?# Benchmarking: how to reproduce every number

Status: runnable methodology. Owner: BenoStreamDB core. Companion docs: BENCHMARKING_PLAN.md (the full plan: datasets, competitor set, metrics schema), RESOURCE_LIMITS.md (memory/CPU knobs), CONCURRENCY.md (writer/reader model).

This document is the reproducibility contract: the exact commands, the environment capture, and the correctness oracles behind every published number. If a number cannot be regenerated from a command in this file, it does not belong in the announcement.

1. The reproducibility contract¶

Every published result must satisfy all five:

One command. The result is produced by a committed script, not a REPL session. The command is recorded next to the number.
Self-describing environment. Each JSON record carries an env block (CPU model, cores, RAM, containerized, GPU list) and the run writes results/hardware_profile.txt (host CPU/RAM, Docker/Compose versions, GPU driver, envelope, dataset, HNSW parameters).
Identical envelope and equivalent configuration. Every participant — server engines and the runner that executes each client, including BenoStreamDB — runs under the same --cpus / --mem cgroup limits, against the same dataset, query workload, k, and index parameters (HNSW m/ef_construction/ef_search, etc.), with recall held at parity. A comparison across different envelopes or different configurations is invalid.
A correctness oracle. The index must not change the answer. Exact indexes are checked against a full scan; ANN indexes are checked for recall. See §7.
One machine state, recorded. See below.

1.1 Machine state and session integrity¶

The machine state is part of the measurement. DIMM timings, a CPU undervolt, frequency-limit changes (amd-pstate/intel_pstate, RAPL), and the kernel move every engine in a table at once — competitor rows included — because they all run on the same host. They do not move everything equally: HNSW/IVF query throughput is latency-bound and moves most, SQL/Parquet scans are bandwidth-bound and move less, and the discrete-GPU rows in §1b do not move at all. A swapped memory kit or a new undervolt therefore invalidates the whole comparison, not one column of it.

The rule that follows: numbers are comparable only within one session, and a session is the set of sections measured in one machine state. A partial re-run cannot be spliced into an older report.

The suite records the state for you. Before/after each section of a sweep:

python benchmarks/session_provenance.py capture --section vector \
    --artifact competitors/results/vector_matrix.json

That writes benchmarks/results/session.json with:

a fingerprint — CPU model, threads, total RAM, DIMM speed/timings (dmidecode -t memory; needs root, otherwise recorded as unknown rather than guessed), governor, frequency driver and limits, RAPL package limit, kernel, and the engine version from Cargo.toml;
calibration probes — random-gather latency (ns/access), triad bandwidth (GB/s), and a fixed fp64 matmul (GFLOPS), sampled once per stamp. The first and last sample bracket the sweep, so thermal or power-limit drift inside a session is visible too;
a section → session map, which generate_summary.py enforces.

generate_summary.py refuses to write the report when stamped sections come from different sessions (exit 2, listing them). --allow-mixed-sessions emits a historical report that labels them instead. Fingerprints are compared on the fields that change a number — CPU, memory, kernel, governor/frequency limits, DIMM config, engine version — never on hostname, PID or timestamps.

Two practical consequences:

After changing RAM timings or the CPU voltage/frequency curve, re-run the whole sweep, competitors included. Re-running only BenoStreamDB produces a table that cannot be read.
Compare versions by interleaving builds on the current state (A/B/A/B), not by diffing a fresh run against stored artifacts from an older state.

Calibration numbers are not results. They exist so a later reader can tell “this column moved because memory latency moved 12%” from “this column moved because the code changed”.

2. Prerequisites¶

Requirement	Why
Docker Engine + Compose v2.30+	`gpus: all` in the GPU override needs Compose ≥ 2.30
NVIDIA Container Toolkit	GPU pass only (`--gpu` / `--both`)
Python 3.10+ with `maturin[zig]`	`--build-wheel` builds the abi3 manylinux wheel
~10 GB free disk	SIFT HDF5 (~525 MB) + runner images (~9 GB GPU)

The runner image installs every competitor client, so no host Python packages are needed for the Docker path.

3. Quick start¶

# CPU baseline: faiss hnswlib lancedb pgvector opensearch benostreamdb
benchmarks/competitors/docker_bench.sh --build-wheel \
    --cpus 8 --mem 16g \
    --dataset sift-128-euclidean --limit 20000 --queries 500

# GPU pass (faiss, benostreamdb, cugraph), same envelope
benchmarks/competitors/docker_bench.sh --gpu --cpus 8 --mem 16g

# Both passes, tagged device=cpu and device=gpu
benchmarks/competitors/docker_bench.sh --both --cpus 8 --mem 16g

--build-wheel runs maturin build --release --zig --compatibility manylinux_2_28 -o dist first. The manylinux tag matters: the runner image is python:3.12-slim (glibc 2.36), so a host-built wheel linked against a newer glibc will not load.

Results land in benchmarks/competitors/results/:

results/{engine}_{dataset}_{device}.json   # one record per engine
results/{engine}_{dataset}_{device}.md     # human-readable sibling
results/rollup.md                          # all engines, one table
results/hardware_profile.txt               # host + envelope + dataset

3.1 Running the whole sweep¶

One command, no setup beyond the prerequisites in §2:

benchmarks/competitors/sweep_full_row.sh

That is the whole contract. The script:

anchors itself at the repo root, because several host harnesses write repo-relative paths;
starts the long-running services (pgvector opensearch qdrant milvus weaviate neo4j clickhouse trino) and waits for health — a fresh clone has none of them;
builds the manylinux wheel once, before it measures anything, so every container installs the same binary and no section can be affected by an edit made mid-run;
measures the nine sections below and quarantines artifacts older than the sweep before each one;
regenerates benchmarks/results/BENCHMARK_REPORT.md as its last step, and exits non-zero if any section is missing from the output.

docker_bench.sh is the per-dataset primitive underneath §3; it is not needed to reproduce the report.

#	Section	What it measures	Artifacts
1	`gpu_cdist`	GPU distance kernels	`results/gpu_cdist.json`
2	`vector_ann`	BenoStreamDB self-characterization, `hnsw` + `hnsw_tq8`, 8 datasets	`ann_benchmarks/results/*`
3	`vector_matrix`	competitor matrix, CPU+GPU pass, enveloped	`competitors/results/*.json`
4	`vector_row`	full row: 11 engines x 8 datasets through the compose stack	`competitors/results/*.json`
5	`graph`	synthetic + `snap-web-google` + `snap-roadnet-ca`	`graph/results/*`
6	`sql`	ClickBench 500k vs DuckDB/DataFusion	`sql/results/*`
7	`beir`	scifact / nfcorpus / arguana: BenoStreamDB vs Tantivy vs OpenSearch	`beir/results/*`
8	`filters`	pathological predicates	`results/production_filters.*`
9	`production`	concurrency sweep, maintenance, skew (§7), soak (§8)	`results/production_*`

Knobs (environment variables)¶

Variable	Default	Meaning
`ONLY`	(all)	comma-separated section names, e.g. `ONLY=vector_row,graph`
`CPUS` / `MEM`	`8` / `16g`	envelope for the containerised section only
`SWEEP_PYTHON`	`.venv/bin/python`	interpreter that measures and summarizes
`BEIR_ENGINES`	`benostreamdb,tantivy,opensearch`	engines for §7, run inside the runner container
`DRY_RUN`	`0`	print the plan, run nothing
`KEEP_GOING`	`1`	continue after a failing section
`SOAK_REPORT`	set by the script	output path for §8; do not override

SWEEP_PYTHON is deliberately not called PYTHON: a conda or pyenv install commonly exports that, and the sweep would then run under an interpreter with no benostreamdb wheel while still passing its first sections.

What the sweep enforces¶

Artifacts older than the sweep are quarantined per section to results/stale/<stamp>-<section>/, so no section mixes two runs.
A section that exits 0 without writing a fresh artifact is a failure, never a stamp — otherwise the previous run’s files are attributed to this session.
Each successful section is stamped through session_provenance.stamp, and generate_summary.py refuses to build a report whose sections span sessions.
generate_summary.py exits 1 and names every section that rendered *No … results found*. A sweep that silently drops a section now fails at the end of the run that caused it.

Per-engine notes¶

The bsdb-bench-runner image is CPU-only, so vector_matrix --both writes _cpu artifacts for both passes; there are no _gpu files unless the GPU overlay (docker-compose.bench.gpu.yml) is in use.
Milvus builds its index explicitly: create_collection(index_params=…) leaves an AUTOINDEX, and AUTOINDEX ignores the HNSW ef parameter entirely, which silently caps recall. The adapter drops the auto index and creates HNSW after ingest (describe_index should report index_type: HNSW).
Weaviate pins the beam width at collection level (dynamic_ef_min == dynamic_ef_max); weaviate-client 4.24.1’s near_vector has no ef argument, so a per-query ef is silently ignored.
run_bench.sh installs exactly one wheel. dist/ can hold both a local maturin build and the manylinux build, and passing the glob to pip fails with ResolutionImpossible before any engine runs.
OpenSearch has no host port. It is only addressable as http://opensearch:9200 on the compose network, so the BEIR comparison must run inside the bench runner (WORKLOAD=beir). Running benchmarks/beir/run.py on the host still works for the two embedded engines (benostreamdb, tantivy), but it cannot reach the server engine and will silently report a two-engine table.

The NeurIPS BigANN and MuSiQue envelopes are not sections of this sweep: they pin different hardware envelopes and are driven separately (§5.1, §5.2).

4. Workload families¶

The same runner image and envelope are reused; --workload selects the family.

4.1 Vector (ANN-Benchmarks)¶

benchmarks/competitors/docker_bench.sh --build-wheel \
    --cpus 8 --mem 16g \
    --dataset sift-128-euclidean --limit 20000 --queries 500 \
    --m 16 --ef-construction 200 --ef-search 200 --k 10

--m, --ef-construction, --ef-search, --metric, --k, --limit, and --queries map 1:1 onto the BenoStreamDB harness flags, so the same configuration runs on both sides without translation.

Like-for-like rule & Iso-Recall evaluation.

Index-Only Isolation: Competitor adapters return vector IDs only (no payload fetch). The rollup therefore reports BenoStreamDB’s pure-index QPS (pure_index_qps), not its full search+row-fetch number, and LanceDB projects only id in its search. Comparing BenoStreamDB’s full-search QPS against a competitor’s search-only QPS conflates index traversal with columnar Parquet decoding and PyArrow RecordBatch construction (§4.1). Full end-to-end query latency is measured separately in §3 and §6.
Iso-Recall Tiering (Option A Parity): ANN search speed is parameterized by beam-search depth (ef_search). To prevent an engine with lower recall from appearing artificially faster, evaluations are normalized across two standard operational tiers: High-Throughput Tier ($\text{Recall}@10 \ge 0.95$, candidate generation) and High-Fidelity Tier ($\text{Recall}@10 \ge 0.99$, top-$k$ ground truth). Within a tier, engines perform equivalent exploration work per query; cross-tier comparisons are evaluated strictly as $(\text{Recall}, \text{QPS})$ Pareto pairs.

4.2 Graph¶

benchmarks/competitors/docker_bench.sh --workload graph \
    --cpus 8 --mem 16g \
    --graph-engines "networkx neo4j cugraph benostreamdb" \
    --algorithm pagerank \
    --graph-nodes 10000 --graph-edges-count 50000

Algorithms: pagerank, connected_components, shortest_path (--source N). A synthetic edge list is generated deterministically (seed 42) and mounted at /opt/bench/data/graph_edges.txt. networkx is the single-threaded reference; cugraph is the GPU reference; neo4j uses the GDS plugin.

4.3 SQL / analytical¶

benchmarks/competitors/docker_bench.sh --workload sql \
    --cpus 8 --mem 16g \
    --sql-engines "duckdb datafusion clickhouse benostreamdb" \
    --sql-rows 500000

A synthetic Parquet table (id, value, category) is generated and mounted at /opt/bench/data/clickbench.parquet; the query is a group-by aggregate.

4.4 Lexical / hybrid (BEIR)¶

The lexical suite lives in benchmarks/beir/ and compares BM25 across three engines on BEIR datasets (scifact, nfcorpus, arguana) under one container envelope: BenoStreamDB (embedded overlay index), Tantivy (embedded Rust IR), and OpenSearch (server-backed, the long-running compose service).

# what the sweep runs, once per dataset
WORKLOAD=beir BEIR_DATASET=scifact \
BEIR_ENGINES=benostreamdb,tantivy,opensearch DEVICE=cpu \
  docker compose -f benchmarks/competitors/docker-compose.bench.yml \
  --project-directory benchmarks/competitors --profile run run --rm bench

The runner writes {engine}_{dataset}_lexical_bm25_{device}.json into benchmarks/beir/results/, which is where generate_summary.py reads §4 from.

BEIR_MODE=hybrid runs run_hybrid.py instead (dense + sparse fused with RRF); its participants are BenoStreamDB and LanceDB, and it is not part of the sweep. See ../benchmarks/beir/.

5. Resource envelope¶

--cpus sets OMP/MKL/OPENBLAS/NUMEXPR_NUM_THREADS; --mem sets the container cgroup deploy.resources.limits.memory. The BenoStreamDB side is constrained by RAYON_NUM_THREADS, BSDB_MAX_CONCURRENCY, and the ingest RAM backpressure (see RESOURCE_LIMITS.md).

RLIMIT_AS is not applied to GPU engines. RLIMIT_AS limits virtual address space, not resident memory. CUDA/RAPIDS managed memory reserves far more virtual address space than physical RAM, so applying the RAM envelope to cugraph (or any --device gpu run) makes the dlopen of libcugraph.so fail with a misleading cannot open shared object file. The container cgroup already enforces the physical RAM envelope, so apply_envelope skips RLIMIT_AS for GPU engines.

Match --cpus/--mem on both sides or the comparison is invalid.

5.1 NeurIPS BigANN (leaderboard-comparable)¶

The BigANN ablation runs under the challenge’s reference envelope — Azure Standard_D8lds_v5, 8 vCPU / 16 GiB — because that envelope determines the cache budgets, not just the thread count:

maturin build --release --out dist
docker build -f benchmarks/Dockerfile.neurips -t bsdb-neurips .
DATASET_DIR=/path/to/bigann \
  docker compose -f benchmarks/docker-compose.neurips.yml run --rm neurips

run_neurips_bigann.py reads the cgroup limits itself (--hardware auto) and derives:

Knob	Derivation
query threads	the CPU limit (env `--cpus`)
`BSDB_CACHE_GB`	half the memory limit
preload budget	the same half, capped
`BSDB_DISK_CACHE`	off — a local `file://` table is served in place, never copied

--hardware d8lds --strict-hardware pins the reference envelope and aborts if the container is not actually constrained, so an oversized host run cannot silently produce a number that looks comparable but is not.

The run is staged so a large corpus is ingested once and only the index is rebuilt per quantizer:

# ingest once (no index), then 3 reindexes + the Pareto grid, in one process
python benchmarks/run_neurips_bigann.py --dataset 10m --stage both --all-modes --sweep

# 100M/1B: ingest in one job, reuse the table for every later run
python benchmarks/run_neurips_bigann.py --dataset 100m --stage ingest
python benchmarks/run_neurips_bigann.py --dataset 100m --stage query --all-modes --sweep

--sweep walks --ef-sweep × --n-probe-sweep (default 32,64,128,256 × 4,8,16,32) against the same warm index, and prints the fastest operating point that clears recall ≥ 0.90 / 0.95 / 0.98 — the row the leaderboard compares against.

Results are written to benchmarks/results/neurips_bigann/:

<tag>.json — every operating point with full provenance: dataset paths, the ground-truth source, engine version + git revision, the envelope, and the grid.
<tag>.md — the Pareto table and the fastest point clearing each recall bar, ready to paste into the paper.
bigann_ledger.jsonl — append-only, one line per run, so numbers can be compared across engine versions.

BenoGraph/paper reads the results from here rather than keeping a second copy.

Two caveats when reading the ablation:

n_probe must scale with the corpus. The IVF list count is sqrt(n)/5 (63 lists at 1M, 632 at 10M), so --n-probe 8 is 12% of the lists at 1M but 1.3% at 10M — recall is dominated by list pruning, not by the quantizer, and all three quantizers return identical neighbours.
At 8 bits over uint8 data, SQ8/TQ8/BQ8 are all effectively lossless. The quantizers only separate above ~0.90 recall, so quote that region, not the coarse-filter end of the curve.

5.2 MuSiQue GraphRAG (multi-hop retrieval)¶

A standalone multi-hop GraphRAG benchmark, separate from the 8-section sweep. It uses the exact corpus, OpenIE graph, and 1,000-question manifest that Zilliz/HippoRAG publish, so the comparison is like-for-like.

Hardware & Memory Envelope (32 GiB vs. BigANN 16 GiB):
While NeurIPS BigANN (§5.1) enforces a strict 16 GiB RAM envelope (Azure Standard_D8lds_v5) for columnar streaming and HNSW indexing, MuSiQue is provisioned with a 32 GiB RAM envelope (--memory=32g in benchmarks/docker-compose.musique.yml and benchmarks/run_musique_docker.sh). The larger memory envelope is strictly required to host the in-memory PyTorch runtime for bge-large-en-v1.5 (1,024-d dense embeddings) and the BGE cross-encoder reranker alongside four native Apache Iceberg tables.

Metric Distinction: MuSiQue evaluates multi-hop retrieval accuracy (Micro Recall@5) across 1,000 2-to-4 hop reasoning questions—it is a retrieval quality benchmark, not a vector throughput (QPS) benchmark.

# headline: engine-native two-stage retrieval, no hosted LLM
python3 benchmarks/run_musique_parity.py --mode f32 --reranker bge --rebuild \
  --corpus-json benchmarks/musique_corpus.json \
  --triplets-json benchmarks/hippo_openie.json \
  --sample-manifest benchmarks/zilliz_manifest.json

# architecture demo: Passages/Entities/Relations/Mentions tables
python3 benchmarks/run_musique_entity_rag.py --mode f32 --reranker bge --rebuild \
  --corpus-json benchmarks/musique_corpus.json \
  --triplets-json benchmarks/hippo_openie.json \
  --sample-manifest benchmarks/zilliz_manifest.json

Headline (1,000 questions, bge-large-en-v1.5, f32, BGE cross-encoder):

Strategy	Micro Recall@5	Description
Hop-1 vector search	0.5600	Direct top-$k$ vector similarity on passage embeddings
Native Personalized PageRank	0.5520	In-engine PPR traversal over passage co-occurrence graph
Two-stage + BGE cross-encoder	0.5943	Vector search + DataFusion graph expansion + cross-encoder

Zilliz’s published Naive RAG baseline is 0.5803; the two-stage row clears it with the whole loop in-engine (vector search + DataFusion graph traversal + local cross-encoder), no hosted LLM and no per-token cost.

The three inputs are git-ignored and reproducible; full entity-level results, data sources, and reproduction steps are documented in MUSIQUE_GRAPHRAG.md.

6. Output schema¶

Each record follows the plan’s schema (§6 of BENCHMARKING_PLAN.md):

{
  "engine": "benostreamdb",
  "workload": "vector",
  "available": true,
  "build_s": 1.23,
  "index_mb": 4.5,
  "recall_at_k": 0.99,
  "qps": 1931.0,
  "pure_index_qps": 2100.0,
  "p50_ms": 0.4,
  "p99_ms": 0.9,
  "env": { "cpu_model": "...", "cores": 8, "ram_gb": 16.0,
           "containerized": true, "gpus": [] }
}

An engine that is not installed or whose server is unreachable emits {"available": false, "error": "..."} instead of crashing, so a partial matrix still produces a complete rollup.

7. Correctness oracles¶

The benchmark numbers are only credible if the index does not change the answer. These are enforced by tests, not by inspection:

Index	Oracle	Test
Scalar bitmap, inverted/BM25, composite	indexed result identical to full scan	`tests/test_differential_index_oracle.rs`
HNSW / IVF / TurboQuant / PQ	exact nearest neighbour appears in indexed top-k (recall)	same file
JSON-path	indexed result identical to full scan; unindexed path falls back to scan	`tests/test_json_path_index.rs`
cuGraph	PageRank matches NetworkX (max abs diff < 1e-5, identical top-5)	`results/cugraph_correct.txt`

Run them with:

cargo test --test test_differential_index_oracle
cargo test --test test_json_path_index

8. Regenerating the rollup and report¶

docker_bench.sh writes results/rollup.md from the JSON records produced during that run (mtime-filtered by RUN_START), so stale results from a previous dataset or device are never double-counted. The rollup uses pure_index_qps for BenoStreamDB (see §4.1).

The narrative report is benchmarks/results/BENCHMARK_REPORT.md; benchmarks/generate_summary.py renders the summary tables.

python benchmarks/session_provenance.py verify          # exit 1 on mixed sessions
python benchmarks/verify_results.py                     # exit 1 on self-contradicting artifacts
python benchmarks/generate_summary.py                   # refuses on mixed sessions
python benchmarks/generate_summary.py --allow-mixed-sessions   # historical report

8.1 Artifact invariants¶

Every defect this benchmark has published was a broken relationship between two artifacts, never a single wrong number: a refined row below its unrefined sibling, a row-path QPS above the index-only QPS it contains, a refined row with no baseline, engines measured at different n/dim/queries/k on one dataset, a qps field carrying the index-only rate. Those are checkable without re-measuring, so benchmarks/verify_results.py checks them and exits non-zero on a violation:

Check	Invariant
`comparability`	every engine on one dataset reports the same `n`, `dim`, `queries`, `k`
`unique cell`	one artifact per `(dataset, device, engine, refine factor)`
`row <= index`	`qps <= pure_index_qps * 1.02` — the row path hydrates the payload
`row measured`	`qps != pure_index_qps` — two columns, two measurements
`timing plausible`	`qps >= 0`, `p50 >= 0`, `p99 >= p50`
`refine baseline`	every refined row has an unrefined sibling
`refine noop (f32)`	refined recall equals unrefined on an uncompressed index
`refine monotone`	refined recall does not drop below unrefined on a quantized index
`refine effective`	a quantized variant’s refine actually moves recall, unless it is already at the ceiling
`machine state`	one `cpu_model` per device across artifacts

generate_summary.py runs the same checks before writing anything and refuses to emit a report when they fail; sweep_full_row.sh runs them as its final gate, so there is no command path that publishes unverified numbers. --skip-verify on the generator exists for archaeology on an old results directory and must not be used for a report that anyone reads.

The generator writes the machine-state fingerprint, the calibration envelope and a per-section session table into the report header, so a reader never has to guess which state a number came from. Artifacts produced before provenance stamping have no session: the report labels them accordingly and treats their mtime table as the only provenance.

9. How to read the numbers¶

The SIFT comparison is 20,000 vectors. The dedicated in-memory engines (FAISS, hnswlib) are optimised for a narrower problem than an Iceberg-native architecture. Do not market the numbers as broad claims. Say: “BenoStreamDB delivers sub-millisecond vector search while maintaining transactional Iceberg tables and persistent overlay indexes” — not “BenoStreamDB is faster than FAISS.”
GPU is slower on small datasets — by design. On the 20k SIFT run the GPU pass is slower than CPU for both engines: fixed PCIe transfer and kernel-launch overheads dominate, the harness measures single-query latency (batch = 1, so there is no GPU parallelism to exploit), and HNSW search is a sequential pointer-chasing walk rather than dense math. BenoStreamDB’s GPU path (benostream-gpu-ann) accelerates index construction, not query. GPU wins at scale: millions of vectors, large batch queries, dense/flat or IVF-PQ search.
Overlay indexes trade build cost and storage for query speed and reconstructibility. The report must state where we lose.
Read the session, not just the section. Before comparing two numbers, check they share a session id in the report header. If they do not, the difference includes a machine-state term and is not a code result. Calibration probes quantify that term: a random gather or triad shift of more than ~5 % between sessions is enough to explain a visible QPS delta on memory-bound workloads.
A powersave governor is a confounder. Check governor and the policy/hw frequency range in the header before treating a small delta as code. Pin the governor (and record it) for any run whose numbers will be published.

10. Troubleshooting¶

Symptom	Cause	Fix
`libcugraph.so: cannot open shared object file`	`RLIMIT_AS` applied to a GPU engine	Fixed in `apply_envelope`; rebuild the base runner image (`docker compose ... build bench bench-gpu`)
Wheel fails to load on the runner (`GLIBC_2.4x not found`)	host-built wheel linked against newer glibc	use `--build-wheel` (manylinux_2_28)
`cudaErrorMemoryAllocation: out of memory` on a 50-series card	cu12 wheels have no sm_120 kernels	the GPU image installs `cugraph-cu13`; `RMM_POOL_SIZE=12GB` leaves headroom for the Wayland compositor
GPU image ships an old harness	`bench-gpu` is `FROM bsdb-bench-runner:latest`	`docker_bench.sh` builds `bench` before `bench-gpu`
`docker compose run` cannot see the GPU	Compose < 2.30 ignores `gpus: all`	upgrade Compose, or use `docker run --gpus all`