Apache Iceberg v3 Explained: Deletion Vectors, Row Lineage and VARIANT

A lakehouse can store petabytes of data and still struggle with a small update. Here is how Apache Iceberg v3 changes the cost of deletes, change tracking and semi-structured data, and how to plan the upgrade from v2.

3 core v3 features Iceberg v2 vs v3 Engine matrix 6-step upgrade plan
Key takeaways
  • Apache Iceberg v3 is a table format specification version, not a new software product.
  • Deletion vectors replace position delete files with one compact bitmap per data file.
  • Row lineage adds _row_id and _last_updated_sequence_number for native change tracking and incremental processing.
  • VARIANT stores semi-structured data in a binary encoding instead of raw JSON strings.
  • The upgrade is one-way, so engine compatibility and workload testing decide the timing.

Imagine a customer table containing hundreds of millions of records. A pipeline receives a batch of corrections affecting a few thousand rows. With copy-on-write, the engine may need to rewrite large data files to apply those changes. With merge-on-read and position deletes, it can avoid rewriting the original files, but readers still have to process separate delete records while planning and scanning queries.

Multiply that workload across frequent updates, merges and compliance-driven deletions. Read overhead, write amplification and file maintenance start consuming resources that should be supporting analytics.

Apache Iceberg v3 addresses several parts of this problem at the table format level. It introduces binary deletion vectors, native row lineage and the VARIANT data type. These changes affect how engines represent deleted rows, identify modified records and store semi-structured data.

For data engineers and architects, the important question is how these mechanics change the cost and complexity of a production data lakehouse architecture.

If you are weighing broader platform patterns, our comparison of data lakehouse vs data mesh architecture sets the context for where Iceberg fits.

01 · THE SPEC

What Is Apache Iceberg v3?

Apache Iceberg v3 is version 3 of the Apache Iceberg table format specification. It is a specification version, not a new software product. Each Iceberg table stores a format-version property in its metadata, and that value decides which features the table can use and which engines can read and write it.

v1Analytic tablesLarge tables on immutable Parquet, Avro and ORC files
v2Row-level deletesPosition and equality delete files, sequence numbers
v3This guideDeletion vectors, row lineage, VARIANT, defaults, new types

Version 2 added row-level updates and deletes for tables built on immutable files. Version 3 builds on that foundation. The Apache Iceberg v3 features most relevant to day-to-day data engineering are:

The v3 specification also adds default column values, nanosecond-precision timestamps and geometry and geography types. This guide focuses on the three features that most directly change how lakehouse pipelines handle updates, change data capture and evolving payloads.

02 · DELETES

Apache Iceberg Deletion Vectors: Record the Delete Without Rewriting the Data File

Iceberg v2 supports row-level deletes through position delete files and equality delete files. Position deletes identify a row through its data file path and row position. Equality deletes identify rows using the values of selected columns.

Both approaches allow changes to immutable data files. However, position delete files create additional objects that readers must match against data files. High-frequency updates can produce a growing collection of delete files and increase scan-planning work.

Iceberg v3 introduces deletion vectors (DVs) as a more efficient representation for position-based row deletions.

A deletion vector stores deleted row positions in a bitmap associated with a specific data file. Each set bit represents a deleted row position. Readers can check the bitmap to determine whether a row should be included in the query result.

Example: deletion vector for one data filerows 0 to 31 · 4 rows deleted
0123456789101112131415 16171819202122232425262728293031
Live row: returned by the querySet bit: row skipped during the scan

How Deletion Vectors Work Inside a Puffin File

Iceberg v3 stores deletion vectors using the deletion-vector-v1 blob format defined by the Puffin specification. Puffin is a file format for storing auxiliary information associated with Iceberg tables. A Puffin file can contain multiple blobs. The Iceberg delete manifest records the Puffin file location, the blob's content offset and its size, along with the referenced data file.

Delete manifest entry

Points to the blob

Puffin file locationBlob content offsetBlob sizeReferenced data file
Puffin file

One file, many blobs

blob · deletion-vector-v1 · data-file-A
blob · deletion-vector-v1 · data-file-B
32-bit keyLocates the relevant bitmap32-bit sub-positionChecked for membership
blob · deletion-vector-v1 · data-file-C

The deletion vector uses Roaring bitmaps to represent row positions efficiently. It supports positive 64-bit positions and divides them into 32-bit keys and 32-bit sub-positions. The reader uses the key to locate the relevant bitmap and checks whether the sub-position is present. This provides an efficient bitmap membership lookup without scanning a separate list of every deleted row.

Position Delete Files vs Deletion Vectors

Iceberg v2 read path

Position delete approach

  1. Find every delete file that applies
  2. Read the delete records
  3. Match file paths and row positions

Delete files keep growing with frequent updates.

Iceberg v3 read path

Deletion vector approach

  1. Load the bitmap for the data file
  2. Check row positions during scanning

At most one deletion vector per data file.

For workloads with frequent small updates, this can reduce delete-processing overhead, improve merge-on-read performance and avoid repeated rewrites of large data files.

Maintenance rule

A v3 snapshot can have at most one deletion vector per data file. Writers must merge new deletions with the existing vector and account for older position deletes when upgrading a table. New position delete files cannot be added to v3 tables.

The benefit depends on workload and engine implementation. Deletion vectors reduce the cost of representing row-level changes, but they do not eliminate compaction, file sizing or metadata maintenance.

Free assessment

Are delete files slowing down your lakehouse?

Get a free review of delete-file growth, compaction cost and merge-on-read performance on your highest-mutation tables.

Book the Free Assessment
03 · ROW LINEAGE

Apache Iceberg Row Lineage: Give Every Row an Identity

Tracking changed records across snapshots has traditionally required additional mechanisms. Teams often maintain timestamps, audit columns, transaction logs or separate change data capture (CDC) pipelines to identify which records have changed.

Iceberg v3 adds native row lineage metadata to the table format. Two fields matter:

Row lineage fieldsEvery v3 row
_row_id
A unique long identifier assigned to a row within the table
_last_updated_sequence_number
The sequence number of the commit that last updated the row
next-row-id
Table-level value used to assign row ID ranges at commit time

When new snapshots are committed, row ID ranges are assigned through metadata inheritance. This avoids rewriting data and manifest files simply because the final row ID range is determined at commit time.

How Iceberg Row Lineage Supports CDC and Incremental Processing

Suppose a downstream warehouse processes a large customer table every hour. Re-reading the entire table to find recent changes is expensive. With row lineage, a consumer can use _last_updated_sequence_number to identify rows changed after its previous processing point, while _row_id provides a stable identifier that helps match the same logical row across snapshots.

A simplified incremental query could look like this:

Incremental querySQL
SELECT customer_id, customer_name, _row_id, _last_updated_sequence_number FROM customer_table WHERE _last_updated_sequence_number > 125;
Try it: move the consumer's checkpoint2 rows to process
125
customer_id_row_id_last_updated_seqStatus
C-10010118Already processed
C-10032124Already processed
C-10054125Already processed
C-10021126Changed: process
C-10043127Changed: process

Illustrative example. Rows keep the same _row_id across snapshots; only the sequence number moves.

Here, 125 represents the consumer's previously processed sequence boundary. The actual implementation must account for snapshot selection, transaction ordering and reliable checkpoint management. Some engines also exclude lineage columns from SELECT *, so name them explicitly in queries.

This makes row lineage useful for Iceberg incremental processing, audit workflows and downstream synchronisation into real-time data pipelines.

What Row Lineage Does Not Do

Row lineage should not be confused with a complete CDC event log. It identifies rows and their last update sequence. It does not independently provide every before-image, deletion event or business-level change event. Consumers still need a strategy for reading snapshots, handling deletes and maintaining checkpoints.

Row lineage gives you
  • A table-unique _row_id per row
  • The sequence number of each row's last update
  • A native way to find rows changed since a checkpoint
You still build
  • Before-images of changed rows
  • Deletion and business-level change events
  • Snapshot reading and checkpoint management

Two further details are worth planning for:

  1. Equality deletes break row identity. When a writer updates a row using equality deletes, the updated row is treated as a new row with a new _row_id. Streaming writers that rely on equality deletes will not preserve a stable identity for updated rows, so do not assume _row_id survives every update mechanism.
  2. Upgraded tables start lineage from the upgrade. When a v2 table is upgraded to v3, existing historical snapshots are not retroactively populated with row IDs. New snapshots establish lineage assignments. Account for this when designing the first incremental load after an upgrade.
04 · VARIANT

Apache Iceberg VARIANT Type: Store Semi-Structured Data Without Flattening Everything

Enterprise pipelines receive data in many shapes. Application events, API responses, product attributes and device telemetry often contain fields that change over time.

A conventional Iceberg schema uses defined types such as structs, lists and maps. These are useful when the data structure is known and governed. But constantly changing payloads can make schema evolution difficult to manage.

Iceberg v3 introduces the VARIANT type for semi-structured values. A VARIANT value can contain objects, arrays and different primitive types. Its structure does not need to be identical across every row. It also supports values such as dates, timestamps, binary data and decimals.

Iceberg VARIANT vs a JSON Text Column

VARIANT is similar to JSON in how it represents flexible data, but its storage model is different. Iceberg uses the binary VARIANT encoding defined by Apache Parquet, where a value is represented through two binary fields.

Incoming payload

Device event (JSON)

{ "device": "pump-07", "temp": 71.4, "tags": ["north", "line-2"], "ts": "2026-11-17T09:12:00Z" }
Stored as VARIANT

Two binary fields

metadataA dictionary of field names, written once0 · device1 · temp2 · tags3 · ts
valueThe encoded object: field IDs from the dictionary plus offsets to typed values (string, decimal, array, timestamp)

The metadata dictionary allows field names to be referenced without repeating the full names throughout every value. Arrays and objects use offsets to locate encoded values, while objects also reference field identifiers in the metadata dictionary. This structure can reduce repeated textual representation and support more efficient processing than treating every payload as an unparsed JSON string.

VARIANT is useful when ingestion needs flexibility. It does not remove the need for data modelling.

Frequently queried fields may still deserve typed columns, statistics and deliberate schema design. Teams should also check how their chosen query engine handles nested values, projection and filtering.

05 · COMPARISON

Iceberg v2 vs v3: What Changes for Data Teams?

format-version: 2 → 3Spec diff
Row-level deletes
Position and equality delete filesBinary deletion vectors for new position-based deletes
Row identity
No native row lineage fields_row_id and update sequence metadata
Incremental processing
Often needs external tracking logicNative lineage metadata supports change identification
Semi-structured data
Structs, maps, lists and engine-specific optionsNative VARIANT type
Delete processing
Separate delete records may increase scan overheadBitmap-based row-position checks
Other additions
Microsecond timestampsNanosecond timestamps, default values, geospatial types

The biggest architectural change is that more information about row-level operations becomes part of the table format itself.

This can simplify pipelines, but the engine must implement the relevant specification features correctly. If you are still deciding between storage approaches, our comparison of data lake vs data warehouse covers the foundations.

06 · ENGINES

Apache Iceberg v3 Engine Compatibility: Spark, Snowflake, Databricks and AWS

Iceberg v3 adoption is not identical across platforms. A table format may define a capability while individual engines support different subsets of its features.

Engine
Deletion vectors
Row lineage
VARIANT
v2 to v3 upgrade
Apache Spark
Snowflake
Databricks
AWS
Documented by the vendor, with conditions belowDepends on Spark version and Iceberg runtime
SP

Apache Spark

The Apache Iceberg project has added v3 support across its Spark integration, including deletion vectors, row lineage work and VARIANT reading. Compatibility depends on the Spark version, Iceberg runtime and specific reader or writer path. Validate both read and write behaviour before enabling v3 in production.

SF

Snowflake

Snowflake documents support for v3 data types, deletion vectors and row lineage across supported managed and externally managed Iceberg tables. The default format version for Iceberg tables in Snowflake remains v2, and in-place upgrades from v2 to v3 are available through ALTER ICEBERG TABLE ... UPGRADE TO ICEBERG VERSION 3. Its Iceberg v3 documentation also specifies limitations around external-engine writes and stream behaviour. Teams on Snowflake should check the current support matrix for their table management model.

DB

Databricks

Databricks documents v3 features for managed Iceberg tables and managed Delta tables with Iceberg reads. Its documented requirements include Unity Catalog and a recent Databricks Runtime (18.0 and above at the time of writing). This should not be interpreted as universal support for every external Iceberg table or engine combination.

AWS

AWS

AWS supports Apache Iceberg v3 features with Apache Spark on Amazon EMR, AWS Glue ETL, Amazon SageMaker Unified Studio notebooks and tables in the AWS Glue Data Catalog, including Amazon S3 Tables. Amazon Redshift can also create and query v3 tables. The AWS Iceberg v3 guide recommends testing upgrades in non-production first and running compaction after an upgrade to consolidate older delete files.

Multi-engine warning

For a multi-engine Apache Iceberg lakehouse, test interoperability at the operation level. A successful read does not prove that another engine can safely write deletion vectors, preserve row IDs during compaction or interpret VARIANT values consistently.

-- one line, one way ALTER TABLE customer_table SET TBLPROPERTIES ('format-version' = '3'); -- no downgrade after this
Iceberg v3 migration

Test every engine before you run this line

We validate reads, writes, compaction and row lineage across your engines on a clone, before any production table is upgraded.

Talk to a Data Engineer
07 · UPGRADE

How to Upgrade Iceberg v2 to v3: Plan Around Your Workload

Iceberg v3 is especially relevant for tables with frequent updates, merges, incremental downstream processing and evolving semi-structured payloads.

Before you change a table's format version, keep one rule in mind: the upgrade is one-way. The Iceberg specification does not support downgrading a table from v3 to v2, and every engine that reads or writes the table must support v3.

Find the right candidate tablesIdentify tables where delete files accumulate, compaction runs frequently or downstream jobs repeatedly scan large datasets to detect changes.Discovery
Measure the baselineRecord current write amplification, query planning time, file counts and maintenance costs.Discovery
Audit every engineList every reader and writer that touches the table and confirm its v3 support, including BI tools and streaming writers.Readiness
Test on a clone firstRun v3 with representative workloads, including concurrent writes, deletes, updates, compaction, snapshot expiration and cross-engine reads.Validation
Validate CDC behaviourVerify how downstream consumers handle row lineage and how existing v2 delete files are migrated.Validation
Point of no return
Upgrade and compactUpgrade during a low-activity window, then run compaction and monitor performance.Production

The right upgrade decision comes from workload evidence and engine compatibility, not the format version alone.

As these pipelines become more distributed, AI agent observability also offers useful lessons in tracing complex execution paths and understanding system behaviour.

08 · SEAFLUX

How Seaflux Helps Data Teams Adopt Apache Iceberg v3

Adopting Apache Iceberg v3 is less about flipping a table property and more about making the format, the engines and the pipelines work together. Seaflux's data engineering services help data teams plan, test and run that change with confidence.

Lakehouse architecture and assessment

Our data lakehouse consulting services review your current tables, delete-file growth, compaction costs and engine mix to decide where Iceberg v3 delivers measurable value.

Delete-file growthCompaction costEngine mix

Iceberg v2 to v3 migration

Cross-engine compatibility testing, clone-based validation and rollback planning, so production tables are never the test environment.

Databricks and Snowflake

Through our Databricks lakehouse consulting and Snowflake implementation services, we configure managed Iceberg tables, Unity Catalog governance and v3 settings.

AWS data engineering

As an AWS Select Consulting Partner, we build Iceberg lakehouses on S3 Tables, Glue, EMR and Redshift, and support cloud migration from legacy platforms.

CDC and incremental pipelines

Incremental processing on row lineage, with checkpoint management and delete handling that downstream analytics teams can trust.

DataOps and table maintenance

Our DataOps services automate compaction, snapshot expiration and pipeline monitoring, so performance gains hold up after go-live.

22%Lower inventory costs in six months
61→81%Forecast accuracy
14 wksTo deliver the lakehouse

See this approach in practice in our Databricks lakehouse and AI demand forecasting case study, where a unified lakehouse gave a UK consumer goods brand faster, more reliable data for planning decisions.

09 · NEXT STEPS

Make the Format Work for Your Data

Apache Iceberg v3 gives data teams more efficient mechanisms for handling row-level mutations, tracking changes and storing flexible payloads. Deletion vectors reduce the overhead associated with separate position delete records. Row lineage gives downstream systems native identifiers and update metadata. VARIANT provides a standard way to represent evolving semi-structured data.

The gains depend on how the table format, compute engine and downstream pipelines work together. Before upgrading, identify one high-mutation table and measure its current delete-file overhead, compaction frequency and incremental processing cost.

Free Data Architecture & Lakehouse Assessment

What is the largest bottleneck in your lakehouse today: frequent updates, CDC maintenance or inconsistent semi-structured data?

Seaflux offers a Free Data Architecture & Lakehouse Assessment to help teams evaluate those bottlenecks and plan a practical lakehouse modernization path.

Delete-file overheadEngine compatibilityCDC readinessv3 upgrade plan

Frequently Asked Questions (FAQ): Get the Answers You Need

Krunal Bhimani

Krunal Bhimani

Business Development Executive

Claim Your No-Cost Consultation!

Let's Connect