Apache Iceberg v3 Explained: Deletion Vectors, Row Lineage and VARIANT
A lakehouse can store petabytes of data and still struggle with a small update. Here is how Apache Iceberg v3 changes the cost of deletes, change tracking and semi-structured data, and how to plan the upgrade from v2.
- Apache Iceberg v3 is a table format specification version, not a new software product.
- Deletion vectors replace position delete files with one compact bitmap per data file.
- Row lineage adds _row_id and _last_updated_sequence_number for native change tracking and incremental processing.
- VARIANT stores semi-structured data in a binary encoding instead of raw JSON strings.
- The upgrade is one-way, so engine compatibility and workload testing decide the timing.
Imagine a customer table containing hundreds of millions of records. A pipeline receives a batch of corrections affecting a few thousand rows. With copy-on-write, the engine may need to rewrite large data files to apply those changes. With merge-on-read and position deletes, it can avoid rewriting the original files, but readers still have to process separate delete records while planning and scanning queries.
Multiply that workload across frequent updates, merges and compliance-driven deletions. Read overhead, write amplification and file maintenance start consuming resources that should be supporting analytics.
Apache Iceberg v3 addresses several parts of this problem at the table format level. It introduces binary deletion vectors, native row lineage and the VARIANT data type. These changes affect how engines represent deleted rows, identify modified records and store semi-structured data.
For data engineers and architects, the important question is how these mechanics change the cost and complexity of a production data lakehouse architecture.
If you are weighing broader platform patterns, our comparison of data lakehouse vs data mesh architecture sets the context for where Iceberg fits.
What Is Apache Iceberg v3?
Apache Iceberg v3 is version 3 of the Apache Iceberg table format specification. It is a specification version, not a new software product. Each Iceberg table stores a format-version property in its metadata, and that value decides which features the table can use and which engines can read and write it.
Version 2 added row-level updates and deletes for tables built on immutable files. Version 3 builds on that foundation. The Apache Iceberg v3 features most relevant to day-to-day data engineering are:
Deletion vectors
A compact, bitmap-based way to record deleted rows.
Jump to section → 02Row lineage
Native row identifiers and update sequence numbers for change tracking.
Jump to section → 03VARIANT type
A native binary type for semi-structured data such as JSON.
Jump to section →The v3 specification also adds default column values, nanosecond-precision timestamps and geometry and geography types. This guide focuses on the three features that most directly change how lakehouse pipelines handle updates, change data capture and evolving payloads.
Apache Iceberg Deletion Vectors: Record the Delete Without Rewriting the Data File
Iceberg v2 supports row-level deletes through position delete files and equality delete files. Position deletes identify a row through its data file path and row position. Equality deletes identify rows using the values of selected columns.
Both approaches allow changes to immutable data files. However, position delete files create additional objects that readers must match against data files. High-frequency updates can produce a growing collection of delete files and increase scan-planning work.
Iceberg v3 introduces deletion vectors (DVs) as a more efficient representation for position-based row deletions.
Position delete files accumulate over time. A deletion vector keeps all deletes for a data file in one bitmap.
A deletion vector stores deleted row positions in a bitmap associated with a specific data file. Each set bit represents a deleted row position. Readers can check the bitmap to determine whether a row should be included in the query result.
How Deletion Vectors Work Inside a Puffin File
Iceberg v3 stores deletion vectors using the deletion-vector-v1 blob format defined by the Puffin specification. Puffin is a file format for storing auxiliary information associated with Iceberg tables. A Puffin file can contain multiple blobs. The Iceberg delete manifest records the Puffin file location, the blob's content offset and its size, along with the referenced data file.
Points to the blob
One file, many blobs
The deletion vector uses Roaring bitmaps to represent row positions efficiently. It supports positive 64-bit positions and divides them into 32-bit keys and 32-bit sub-positions. The reader uses the key to locate the relevant bitmap and checks whether the sub-position is present. This provides an efficient bitmap membership lookup without scanning a separate list of every deleted row.
Position Delete Files vs Deletion Vectors
Position delete approach
- Find every delete file that applies
- Read the delete records
- Match file paths and row positions
Delete files keep growing with frequent updates.
Deletion vector approach
- Load the bitmap for the data file
- Check row positions during scanning
At most one deletion vector per data file.
For workloads with frequent small updates, this can reduce delete-processing overhead, improve merge-on-read performance and avoid repeated rewrites of large data files.
A v3 snapshot can have at most one deletion vector per data file. Writers must merge new deletions with the existing vector and account for older position deletes when upgrading a table. New position delete files cannot be added to v3 tables.
The benefit depends on workload and engine implementation. Deletion vectors reduce the cost of representing row-level changes, but they do not eliminate compaction, file sizing or metadata maintenance.
Are delete files slowing down your lakehouse?
Get a free review of delete-file growth, compaction cost and merge-on-read performance on your highest-mutation tables.
Apache Iceberg Row Lineage: Give Every Row an Identity
Tracking changed records across snapshots has traditionally required additional mechanisms. Teams often maintain timestamps, audit columns, transaction logs or separate change data capture (CDC) pipelines to identify which records have changed.
Iceberg v3 adds native row lineage metadata to the table format. Two fields matter:
- _row_id
- A unique long identifier assigned to a row within the table
- _last_updated_sequence_number
- The sequence number of the commit that last updated the row
- next-row-id
- Table-level value used to assign row ID ranges at commit time
When new snapshots are committed, row ID ranges are assigned through metadata inheritance. This avoids rewriting data and manifest files simply because the final row ID range is determined at commit time.
How Iceberg Row Lineage Supports CDC and Incremental Processing
Suppose a downstream warehouse processes a large customer table every hour. Re-reading the entire table to find recent changes is expensive. With row lineage, a consumer can use _last_updated_sequence_number to identify rows changed after its previous processing point, while _row_id provides a stable identifier that helps match the same logical row across snapshots.
A simplified incremental query could look like this:
| customer_id | _row_id | _last_updated_seq | Status |
|---|---|---|---|
| C-1001 | 0 | 118 | Already processed |
| C-1003 | 2 | 124 | Already processed |
| C-1005 | 4 | 125 | Already processed |
| C-1002 | 1 | 126 | Changed: process |
| C-1004 | 3 | 127 | Changed: process |
Illustrative example. Rows keep the same _row_id across snapshots; only the sequence number moves.
Here, 125 represents the consumer's previously processed sequence boundary. The actual implementation must account for snapshot selection, transaction ordering and reliable checkpoint management. Some engines also exclude lineage columns from SELECT *, so name them explicitly in queries.
This makes row lineage useful for Iceberg incremental processing, audit workflows and downstream synchronisation into real-time data pipelines.
What Row Lineage Does Not Do
Row lineage should not be confused with a complete CDC event log. It identifies rows and their last update sequence. It does not independently provide every before-image, deletion event or business-level change event. Consumers still need a strategy for reading snapshots, handling deletes and maintaining checkpoints.
- A table-unique _row_id per row
- The sequence number of each row's last update
- A native way to find rows changed since a checkpoint
- Before-images of changed rows
- Deletion and business-level change events
- Snapshot reading and checkpoint management
Two further details are worth planning for:
- Equality deletes break row identity. When a writer updates a row using equality deletes, the updated row is treated as a new row with a new _row_id. Streaming writers that rely on equality deletes will not preserve a stable identity for updated rows, so do not assume _row_id survives every update mechanism.
- Upgraded tables start lineage from the upgrade. When a v2 table is upgraded to v3, existing historical snapshots are not retroactively populated with row IDs. New snapshots establish lineage assignments. Account for this when designing the first incremental load after an upgrade.
Apache Iceberg VARIANT Type: Store Semi-Structured Data Without Flattening Everything
Enterprise pipelines receive data in many shapes. Application events, API responses, product attributes and device telemetry often contain fields that change over time.
A conventional Iceberg schema uses defined types such as structs, lists and maps. These are useful when the data structure is known and governed. But constantly changing payloads can make schema evolution difficult to manage.
Iceberg v3 introduces the VARIANT type for semi-structured values. A VARIANT value can contain objects, arrays and different primitive types. Its structure does not need to be identical across every row. It also supports values such as dates, timestamps, binary data and decimals.
Iceberg VARIANT vs a JSON Text Column
VARIANT is similar to JSON in how it represents flexible data, but its storage model is different. Iceberg uses the binary VARIANT encoding defined by Apache Parquet, where a value is represented through two binary fields.
Device event (JSON)
Two binary fields
The metadata dictionary allows field names to be referenced without repeating the full names throughout every value. Arrays and objects use offsets to locate encoded values, while objects also reference field identifiers in the metadata dictionary. This structure can reduce repeated textual representation and support more efficient processing than treating every payload as an unparsed JSON string.
Frequently queried fields may still deserve typed columns, statistics and deliberate schema design. Teams should also check how their chosen query engine handles nested values, projection and filtering.
Iceberg v2 vs v3: What Changes for Data Teams?
The biggest architectural change is that more information about row-level operations becomes part of the table format itself.
This can simplify pipelines, but the engine must implement the relevant specification features correctly. If you are still deciding between storage approaches, our comparison of data lake vs data warehouse covers the foundations.
Apache Iceberg v3 Engine Compatibility: Spark, Snowflake, Databricks and AWS
Iceberg v3 adoption is not identical across platforms. A table format may define a capability while individual engines support different subsets of its features.
Apache Spark
The Apache Iceberg project has added v3 support across its Spark integration, including deletion vectors, row lineage work and VARIANT reading. Compatibility depends on the Spark version, Iceberg runtime and specific reader or writer path. Validate both read and write behaviour before enabling v3 in production.
Snowflake
Snowflake documents support for v3 data types, deletion vectors and row lineage across supported managed and externally managed Iceberg tables. The default format version for Iceberg tables in Snowflake remains v2, and in-place upgrades from v2 to v3 are available through ALTER ICEBERG TABLE ... UPGRADE TO ICEBERG VERSION 3. Its Iceberg v3 documentation also specifies limitations around external-engine writes and stream behaviour. Teams on Snowflake should check the current support matrix for their table management model.
Databricks
Databricks documents v3 features for managed Iceberg tables and managed Delta tables with Iceberg reads. Its documented requirements include Unity Catalog and a recent Databricks Runtime (18.0 and above at the time of writing). This should not be interpreted as universal support for every external Iceberg table or engine combination.
AWS
AWS supports Apache Iceberg v3 features with Apache Spark on Amazon EMR, AWS Glue ETL, Amazon SageMaker Unified Studio notebooks and tables in the AWS Glue Data Catalog, including Amazon S3 Tables. Amazon Redshift can also create and query v3 tables. The AWS Iceberg v3 guide recommends testing upgrades in non-production first and running compaction after an upgrade to consolidate older delete files.
For a multi-engine Apache Iceberg lakehouse, test interoperability at the operation level. A successful read does not prove that another engine can safely write deletion vectors, preserve row IDs during compaction or interpret VARIANT values consistently.
Test every engine before you run this line
We validate reads, writes, compaction and row lineage across your engines on a clone, before any production table is upgraded.
Talk to a Data EngineerHow to Upgrade Iceberg v2 to v3: Plan Around Your Workload
Iceberg v3 is especially relevant for tables with frequent updates, merges, incremental downstream processing and evolving semi-structured payloads.
Before you change a table's format version, keep one rule in mind: the upgrade is one-way. The Iceberg specification does not support downgrading a table from v3 to v2, and every engine that reads or writes the table must support v3.
The right upgrade decision comes from workload evidence and engine compatibility, not the format version alone.
As these pipelines become more distributed, AI agent observability also offers useful lessons in tracing complex execution paths and understanding system behaviour.
How Seaflux Helps Data Teams Adopt Apache Iceberg v3
Adopting Apache Iceberg v3 is less about flipping a table property and more about making the format, the engines and the pipelines work together. Seaflux's data engineering services help data teams plan, test and run that change with confidence.
Lakehouse architecture and assessment
Our data lakehouse consulting services review your current tables, delete-file growth, compaction costs and engine mix to decide where Iceberg v3 delivers measurable value.
Iceberg v2 to v3 migration
Cross-engine compatibility testing, clone-based validation and rollback planning, so production tables are never the test environment.
Databricks and Snowflake
Through our Databricks lakehouse consulting and Snowflake implementation services, we configure managed Iceberg tables, Unity Catalog governance and v3 settings.
AWS data engineering
As an AWS Select Consulting Partner, we build Iceberg lakehouses on S3 Tables, Glue, EMR and Redshift, and support cloud migration from legacy platforms.
CDC and incremental pipelines
Incremental processing on row lineage, with checkpoint management and delete handling that downstream analytics teams can trust.
DataOps and table maintenance
Our DataOps services automate compaction, snapshot expiration and pipeline monitoring, so performance gains hold up after go-live.
See this approach in practice in our Databricks lakehouse and AI demand forecasting case study, where a unified lakehouse gave a UK consumer goods brand faster, more reliable data for planning decisions.
Make the Format Work for Your Data
Apache Iceberg v3 gives data teams more efficient mechanisms for handling row-level mutations, tracking changes and storing flexible payloads. Deletion vectors reduce the overhead associated with separate position delete records. Row lineage gives downstream systems native identifiers and update metadata. VARIANT provides a standard way to represent evolving semi-structured data.
The gains depend on how the table format, compute engine and downstream pipelines work together. Before upgrading, identify one high-mutation table and measure its current delete-file overhead, compaction frequency and incremental processing cost.
What is the largest bottleneck in your lakehouse today: frequent updates, CDC maintenance or inconsistent semi-structured data?
Seaflux offers a Free Data Architecture & Lakehouse Assessment to help teams evaluate those bottlenecks and plan a practical lakehouse modernization path.
Frequently Asked Questions (FAQ): Get the Answers You Need
What is new in Apache Iceberg v3?
Apache Iceberg v3 adds binary deletion vectors, native row lineage, the VARIANT type for semi-structured data, default column values, nanosecond timestamps and geometry and geography types. Deletion vectors, row lineage and VARIANT have the biggest impact on everyday lakehouse pipelines.
What are deletion vectors in Apache Iceberg?
Deletion vectors are compact Roaring bitmaps, stored in Puffin files, that mark which row positions in a data file are deleted. They replace position delete files in v3 tables, so readers check one bitmap per data file instead of matching many separate delete records.
Does Apache Iceberg v3 support CDC?
Iceberg v3 row lineage supports change data capture by giving each row a _row_id and a _last_updated_sequence_number. Consumers can query rows changed after a known sequence number. It is not a full CDC event log, so pipelines still need checkpoint management and delete handling.
What is the difference between Iceberg v2 and v3?
Iceberg v2 introduced row-level deletes through position and equality delete files. Iceberg v3 replaces new position deletes with deletion vectors, adds native row lineage for change tracking and introduces the VARIANT type, along with new timestamp, default value and geospatial capabilities.
Can you downgrade an Iceberg table from v3 to v2?
No. The Apache Iceberg specification does not support downgrading format versions. Test the upgrade on a clone first and confirm that every engine reading or writing the table supports v3 before upgrading production tables.
Should I upgrade my Iceberg tables to v3?
Upgrade when a table has frequent updates and merges, needs incremental processing or stores evolving semi-structured data, and when every engine touching it supports v3. For stable, append-only tables with broad engine dependencies, staying on v2 for now can be the safer choice.

Krunal Bhimani
Business Development Executive