Category: polars

Iceberg on a single node is coming together

For a long time, whenever I tried to use Iceberg outside the JVM ecosystem, something I needed seemed to be waiting for the next release. It took longer than I hoped, but lately I have been able to get much further with the engines I use.

chDB can authenticate to OneLake with a token, Polars now runs all 22 queries in this benchmark, and DuckDB’s cache feels like absolute magic on the second pass. I still ran into bugs, but I could run both reads and writes across several engines against the same Iceberg catalog.

I took two of my notebooks and port them to GitHub Actions as python script across DuckDB, Polars, chDB, LakeSail, Spark and Daft. One runs analytical queries; the other reads CSVs and writes Iceberg tables. I wanted to see how each handled the workloads on a small 4-core runner.

The data lives in OneLake. Reading through its Iceberg REST catalog is already available; the write support I used is still in private preview.

Both benchmarks run on a public 4 vCPU / 16 GB runner. The engines connect to the catalog either directly or through pyiceberg. OIDC federation provides a token per job, so there is no client secret to store. The same approach should work with other Iceberg REST catalogs, provided the engine supports their storage and authentication.

The code, the raw JSON of every run and the charts are in djouallah/lakehouse_benchmark.

Reading: TPC-H-like queries at SF10

The tables are generated with tpchgen, uploaded as Parquet and registered with pyiceberg’s add_files. DuckDB and LakeSail could have written them just fine, but I wanted to stay neutral. Each engine attaches the catalog, runs the 22 queries, then runs the same 22 again immediately. Those are the cold and warm passes below; catalog attach time is excluded. The chart and the table are the mean of three runs on 20 and 21 September 2026. This is a small single-node comparison, not an official TPC-H benchmark.

EngineVersionColdWarm
DuckDB2.0.0 dev35.9s21.3s
Polars2.0.0-rc.292.1s88.6s
chDB4.4.0125.4s104.5s
LakeSail0.7.1159.9s161.6s
Spark-OSS4.1.3484.9s454.9s

Spark open source, not to be confused with Fabric Spark, is a single-node JVM on 4 cores here. Thanks to AI, running Spark locally is no longer a scary experience. Unfortunately Daft is missing: at 0.7.25 it runs 16 of the 22 queries, because l_extendedprice * (1 - l_discount) overflows its decimal precision ceiling of 38 (Daft#7532). I like Daft and would love to include it in the full query comparison once this is fixed. It does complete the ETL workload below.

Writing: Light ETL

1000 daily AEMO CSV files, 52 GB, are landed once in the lakehouse Files/ section. Each engine reads all of them, filters, casts and writes one Iceberg table of 149,146,763 rows through the same catalog. DuckDB, LakeSail and Spark have their own Iceberg writer. chDB and Polars stream Arrow batches into pyiceberg. Daft has its own writer over a pyiceberg table.

EngineLoad
Polars470.8s
DuckDB497.4s
LakeSail684.5s
Daft757.9s
chDB878.1s
Spark-OSS982.2s

Every engine ends with the same row count. Load time includes recreating the table, reading the CSVs, transforming, writing and committing; session setup and catalog attach are timed separately. There is one load pass per engine, and the chart and the table are the mean of three runs. Snapshot isolation and concurrent writes will have to wait for another blog. The Iceberg write path used here is in private preview.

Random learnings, in no particular order

  • DuckDB needed AZURE_TRANSPORT_OPTION_TYPE=curl on this Linux runner. Without it the Iceberg attach succeeds and every data-file read fails with a message that looks exactly like a missing credential. Setting the transport fixed it. I had not seen this in the Fabric notebook.
  • chDB returned 41 rows for Q13, everybody else 42. ClickHouse defaults join_use_nulls=0 and fills unmatched outer-join cells with default values instead of NULL. The row-count smoke test caught this, and the benchmark now sets join_use_nulls=1.
  • The catalog cache settings do not all mean the same thing. I configured a 15-minute lifetime, but LakeSail’s “table cache” caches the table listing, not the loaded table. It still loads table metadata on every statement, adding REST requests even on the warm pass (sail#2629).
  • Daft’s six query failures come from decimal arithmetic. CTEs, EXISTS and correlated subqueries all work. It also rejects backticks outright, which looked like a total dialect failure until I read the error. Separately, its Azure URI parser drops the container on OneLake hosts; the bench works around it with az:// paths. The proposed fix is Daft#7533.
  • Nothing is partitioned, on purpose. The pyiceberg streaming append used here only supports unpartitioned tables. A first version that materialised batches to partition by year took the 16 GB runner down at 100 files. So every engine writes the same unpartitioned shape and year stays a plain column.

Spark, and why it has no native accelerator here

Spark-OSS is 4.1.3 with Iceberg 1.11 and hadoop-azure 3.4.2. I also looked at Comet and Gluten/Velox, but could not use either for this benchmark.

  • Comet supports Spark 4.1 and ships a native Iceberg reader, but its readable schemes are file, s3, s3a, gs, oss. An abfss table is declined at planning time and runs on the JVM as if the plugin were not there. I left Comet out because it could not accelerate these scans. Filed as datafusion-comet#6058.
  • Gluten/Velox has a Velox ABFS connector, but I could not find a published package for Spark 4.x to test (This basically what Fabric NEE uses)

Conclusion

All six engines completed the CSV-to-Iceberg load. Five completed all 22 read queries; Daft completed 16. Being able to try this many engines against one catalog, with the data staying in OneLake, is what I find exciting. chDB’s token support and Polars running all 22 SQL queries give me more options for the notebooks I already use.

DuckDB was fastest on the reads, and on the load it traded first place with Polars from run to run. DuckDB’s read total fell from 36s to 21s on the second pass, about 40%. The other engines improved less, and LakeSail’s warm pass came out slightly slower than its cold one. That makes repeated reads worth testing separately from a one-pass load. It does not tell us exactly what each engine cached or how many bytes it fetched; chDB also has a filesystem cache configured here.

For my ETL workload, the cache does not matter: read the new files, transform them and write the table.

If you want to see how much things have changed, have a look at these TPC-H SF10 results from four years ago.

The timings above come from the JSON under results/. To run the benchmarks against your own workspace, follow RUN.md for the Entra app registration and tenant settings. The write benchmark requires access to the private preview used here coming soon.

How far Python alone can take you on Delta

1. delta-rs is an ACID Delta writer

delta-rs implements the Delta Lake protocol natively. merge, update, and delete go through optimistic concurrency control on every commit. No external coordinator, no catalog service. Two writers race for the same version of the log, one wins, the other retries.

All you need is a path. No metastore to provision, no catalog endpoint, no JDBC connection, no warehouse to wake up. A folder on disk (or on ADLS / S3 / GCS) is the whole interface.

Setup: B is a Delta table being fed a series of CSV batches (batch_001.csv, batch_002.csv, …). Each merge should ingest only files B hasn’t seen yet.

A naming note: the project is delta-rs but the Python package is deltalake (pip install deltalake). On Fabric, stick with what’s preinstalled — Python notebooks already ship with deltalake and OneLake access configured.

From the notebook:

# Bootstrap target B with batch_001 already ingested
write_deltalake(Target_PATH, pa.table({...}), mode="overwrite")
vB = DeltaTable(Target_PATH).version()   # v0
# Compute the rows to ingest from the target's current state
con.sql(f"ATTACH '{Target_PATH}' AS tgt (TYPE delta, VERSION {vB});")
our_rows = con.sql("""
    SELECT s.id, s.value, parse_filename(s.filename) AS filename
    FROM read_csv_auto('source_csv/*.csv', filename=true) s
    WHERE parse_filename(s.filename) NOT IN (SELECT DISTINCT filename FROM tgt)
""").arrow()
# → 80 new rows from batch_002..005
# First merge: 80 inserts, commits cleanly
DeltaTable(Target_PATH).merge(
    source=our_rows,
    predicate="t.filename = s.filename",
    source_alias="s", target_alias="t",
).when_not_matched_insert_all().execute()
# Same merge re-run: 0 inserts. The predicate is idempotent.
DeltaTable(Target_PATH).merge(...).when_not_matched_insert_all().execute()

Two commits, both correct. The second run does nothing because the predicate already sees the rows. The transaction model travels with the table itself: move the folder, open it from another machine, and the next writer continues from the last commit.

write_deltalake(mode="append") and write_deltalake(mode="overwrite") are blind on purpose. Blind append means N concurrent appenders all succeed and the result is the union of their rows — exactly what you want for event streams or log ingestion. Blind overwrite means the new data wins and whatever was there is gone — what you want when the writer is the authoritative source for the table. OCC only kicks in for operations that actually read the target (merge, update, delete), since those are the only ones where a concurrent change can invalidate what you just computed.

2. I want the full read-to-write transaction, Python API is fine

A common pattern: DuckDB or Polars reads, transforms, and hands an Arrow table to delta-rs to commit. The notebook above is exactly that shape — DuckDB computes “filenames not yet in B” and delta-rs merges the result.

Inside delta-rs, OCC still works. What it cannot see is the read on the other side of the engine boundary. delta-rs knows about the merge it is about to commit; it does not know that DuckDB read B at version vB thirty seconds ago.

Carry the snapshot across the boundary by pinning both sides to the same version:

vB = DeltaTable(Target_PATH).version()
import duckdb
con = duckdb.connect()
con.sql(f"ATTACH '{Target_PATH}' AS tgt (TYPE delta, VERSION {vB});")
our_rows = con.sql("SELECT ...").arrow()
DeltaTable(Target_PATH, version=vB).merge(   # ← pinned
    source=our_rows,
    predicate="t.filename = s.filename",
    source_alias="s", target_alias="t",
).when_not_matched_insert_all().execute()

The OCC check now compares against vB instead of HEAD. If another process touched B in the meantime — say a parallel job deleted batch_001.csv — the pinned merge raises:

Failed to commit transaction: Commit failed: a concurrent transaction deleted data this operation read.

Catch it, recompute the diff against fresh state, retry. On the Polars side, pl.read_delta(path, version=vB) accepts the same pin, so the pattern works for any reader that exposes versioned reads.

The pin is just a number. No new infrastructure, no shared coordinator, still path-based.

3. I don’t want the Python API, I want SQL only

If you would rather write SQL — say, drive the pipeline from dbt — your options on Delta today are Spark and Fabric Data Warehouse. Both have supported dbt adapters and work great in production. I have to admit, I was hoping DuckDB would fill that gap, since it is a database and SQL-level transactions are what you expect from a database. The market went the other way: investment is going into catalog-based lakehouse formats (DuckLake, Iceberg), and the DuckDB Delta writer that does exist is tied to Unity Catalog and limited to blind appends. I don’t see them investing in a file-based conflict resolver any time soon 🙂 Lakesail seems interested in this use case, but it is still too early to call.

Takeaway

I personally use delta-rs for CSV ingestion, appends, and recording results from high-concurrency performance tests — it is fast, cheap, and bullet-proof in those scenarios. The open source maintainers are very helpful and care deeply about the product, as they use it themselves in production. But it is not the right tool for every case; Data Warehouse and Spark are more appropriate for complex workloads. With time you intuitively pick the tool that makes sense for a particular job and how much compute you can spend. None of that has to be an either/or: at the end of the day it is a lakehouse, and the whole concept of a lakehouse is having the option to choose the engine. That option matters — if we say only one engine (open source or not) is blessed for writes, then there is no point in the concept of a lakehouse.

Notebook: https://github.com/djouallah/Fabric_Notebooks_Demo/blob/main/TableFormat/delta/occ.ipynb

Thanks Raki for keeping me honest:)

Thanks to Ion for explaining how version worked when doing merge: https://www.linkedin.com/in/ionkoutsouris/

Edit : how about Spark

Thanks to Frithjof for explaining Spark behaviour : The merge fixes one snapshot at transaction start (current HEAD = post-delete) and uses it for both its scan and its conflict check. Internally consistent — but bound to HEAD-at-merge-start, which Spark chose, not to the state our read saw, same behaviour when using delta_rs with a lazy dataframe : https://github.com/djouallah/Fabric_Notebooks_Demo/blob/main/TableFormat/delta/occ_spark.ipynb