ADR-010: Iceberg V3 Writes Must Be the Spec, and a Foreign Reader Is the Acceptance Test

Status: accepted (v0.16.0). Implemented across puffin.rs, iceberg/writer.rs, iceberg/manifest.rs, iceberg/delete.rs, manifest/types.rs, metadata.rs, table/maintenance.rs, and merge.rs. Change history: see the [Unreleased] / 0.16.0 entries in CHANGELOG.md. Catalog-side decisions from the same programme are in ADR-009.

---

1. Context

BenoStreamDB’s portability claim rests on writing ordinary Apache Iceberg metadata: v{N}.metadata.json, Avro manifest lists, Avro data/delete manifests, Parquet segments, Puffin sidecars. Two forces made a decision necessary.

Force 1 — a private dialect is a silent data-loss risk. The engine had begun encoding V3 concepts in shapes only it understood: the manifest Avro metadata hardcoded format-version = "2"; the deletion-vector Puffin blob used a foreign type identifier (apache-datasketches-theta-v1) with a raw roaring payload; delete entries used a custom content = 3 code; and the V3 manifest schema never emitted fields 143–145. A foreign engine opening such a table has two choices and both are bad: reject it, or trust the metadata and return rows the writer deleted.

Force 2 — the existing verification could not detect this class of bug. PyIceberg 0.12 is V2-centric, so it validates the V2 surface and ignores V3 semantics. Worse, BenoStreamDB’s own reader applies every deletion vector attached to a manifest entry and never consults referenced_data_file, so it is structurally incapable of noticing a vector that no external engine can match. “Our reader agrees with our writer” is not evidence.

2. Decision

D1 — Write the published V3 spec, never a private encoding. Deletion vectors use blob type deletion-vector-v1 and the spec envelope (4-byte big-endian length, magic D1 D3 39 64, portable 64-bit roaring bitmap, 4-byte big-endian CRC-32). Manifest metadata carries the table’s own format-version. Delete entries use content = 1 (POSITION_DELETES) with file_format = "PUFFIN" and populated referenced_data_file / content_offset / content_size_in_bytes.

Rejected: keeping the custom content = 3 encoding and documenting it. It is strictly less work, but it makes every V3 table engine-specific, which is the one thing an open table format exists to prevent. The legacy code is still accepted on read so tables written by earlier builds keep resolving.

D2 — V2 is frozen, not migrated. Every V3 behavior is gated on format_version >= 3. V2 tables keep Parquet position deletes and their manifests still declare format-version = 2.

Rejected: silently upgrading tables on write. It would change the delete representation under operators who never asked for it, and V2 remains the interchange format most clusters actually accept.

D3 — A foreign reader is the acceptance criterion, and it must not share code with the engine. Conformance means apache/iceberg-rust, Spark, or Trino opens the table and sees the exact row set. The Rust oracle is a separate workspace that depends on iceberg only; it communicates with the engine exclusively through on-disk artifacts.

Rejected: asserting against PyIceberg, or against our own reader. Both already passed while all three of the defects listed in §3 were live. Code sharing is what made them invisible.

D4 — Reject rather than repair. A deletion vector with a bad length, magic, or CRC is a typed error (BenoStreamError::DeletionVectorDeserialize), not a warning with a best-effort bitmap.

Rejected: tolerating malformed vectors. A silently-short delete list resurrects deleted rows; a failed query is the better failure.

3. Consequences

Positive. A format-version = 3 table written by BenoStreamDB is readable by iceberg-rust 0.11 and by Spark/Trino, including after deletions. V3 lineage is publishable: each snapshot carries a row range so a reader can assign stable _row_id values. The engine’s delete representation is now the interchange format rather than a local convention.

Negative and accepted.

• CRC-32 is hand-rolled (IEEE 802.3, ~15 lines) instead of adding a dependency, because deny.toml audits every crate in the graph.

• Positions above u32::MAX are an error, not a 64-bit position. The engine’s row positions are u32; a table with more than 4 billion rows in one data file is out of scope, and the failure is loud.

• Table::write_mode() stays "buffered" on V3. Distributed Spark/Trino task staging cannot reserve disjoint _row_id ranges without driver-side allocation. Safety comes from ADR-008’s PendingCommit machine, not from a staged-write contract.

• _row_id is not in current_schema(). It is a metadata column by spec; lineage lives in next-row-id and the snapshot row range. Tooling that expects to SELECT _row_id will not find it there.

• Runtime pinning is bounded by what Iceberg publishes. iceberg-spark-runtime exists only for Spark minor series 3.4, 3.5, 4.0 and 4.1 — there is no 4.2 artifact, so Spark 4.2 has no supported pairing. The JVM matrix therefore pins Spark 3.5.2 (3.5_2.12, Java 17) and 4.0.x (4.0_2.13, Java 21) on Iceberg 1.11; 1.6 predates V3 vectors.

• The JVM matrix runs weekly, not per-PR. A Spark submit is a JVM start plus a ~200 MB Maven resolve. The trade is accepted because the format only moves when src/core/iceberg/** or metadata.rs changes, which the workflow’s path filter watches.

Risk this decision accepts. A bug in the V3 codec is now a bug in a format other engines depend on; there is no longer a private encoding to fall back to. The mitigation is the three independent readers in §4, not the engine’s own test suite.

4. How the Decision Is Enforced

Gate	Enforces	Cadence
tests/test_iceberg_v3_conformance.rs	D1, D2 — manifest metadata/schema shape, parsed by a hand-written Avro container reader rather than apache_avro, so a wrong key name cannot pass by construction	every push
tests/iceberg_rust_oracle	D3 — iceberg-rust opens the table, asserts format version, next_row_id, snapshot row range, and the exact scanned row set after a vector	every push
.github/workflows/iceberg-interop.yml	D3 — Spark 3.5/4.0 and Trino 482/484 read the table, including V3 lineage columns	weekly + on table-format changes
puffin.rs unit tests	D4 — round-trip plus rejection of bad length, magic, and CRC	every push

This ADR should be revisited if: Iceberg publishes a Spark runtime for a new minor series (re-pin the matrix), apache/iceberg-rust starts matching vectors by a different key (re-check D1’s path normalization), or the engine gains 64-bit row positions (D4’s u32 bound must be lifted).