Skip to main content
Data Depot can deliver the Locations and Waybills products as Parquet to your Azure or Amazon S3 landing zone. The Cedar Databricks notebook turns one delivery generation into a current-state Delta table in your workspace.
CDC delivery is currently an EU pilot for carriers on eu-north-1:arms-prod. Postgres JDBC and public API access remain available independently for other entitled carriers.
Do not point Auto Loader directly at the Cedar Parquet folders. Cedar uploads each Parquet object before its manifest.json. The manifest is the commit signal. Reading the Parquet path directly can process an incomplete or abandoned batch.

​
Before you start

You need:
  • a Databricks workspace with Delta Lake and access to create tables;
  • a Unity Catalog external location, volume, or other Databricks storage configuration that can read your landing zone;
  • a live Data Depot CDC delivery; and
  • separate read access for Databricks. Data Depot stores the credential Cedar uses to write the delivery, but does not share that secret with Databricks.
Data Depot currently supports CDC delivery for Locations and Waybills. Other Postgres provisions are pull-only and do not produce this Parquet layout. The initial snapshot is currently delivered as one Parquet object. Contact Cedar before onboarding a very large carrier dataset so the first snapshot can be checked against the current delivery-worker capacity.

​
1. Create and verify the delivery

1

Create a subscription

In Data Depot, open Delivery, select Subscribe to CDC, then choose Locations or Waybills and an Azure or S3 destination.
2

Wait for the snapshot

Open the subscription and wait until Snapshot delivered has a timestamp. A live subscription can also have newer change batches.
3

Record the generation details

Copy the Databricks base path shown under Databricks ingestion. Data Depot converts Azure container URLs to abfss:// and includes the destination prefix, subscription ID, and generation. The same panel provides the notebook download and this guide.
The underlying relative generation path has this form:
Data Depot assembles it with your destination and optional prefix:

​
2. Import the notebook

On the subscription detail page, select Download notebook. You can also download it from Databricks in Data Depot. In your Databricks workspace, import cedar_product_cdc_to_delta.py as a source file. The notebook has four parameters: Use a new target_table when you load a generation for the first time. Multiple Cedar targets can share one state_table.

​
3. Run and schedule it

Run the notebook once interactively. It will:
  1. require the committed snapshot manifest;
  2. validate and load the snapshot;
  3. expand the product fields from payload_json into columns;
  4. find committed change manifests;
  5. ignore already-applied rows from overlapping delivery ranges;
  6. keep the newest event for each product key in a batch; and
  7. update, insert, or delete rows with a sequence-guarded Delta MERGE.
After the first successful run, create a Databricks job using the same parameters. Schedule it at or after the delivery interval selected in Data Depot. The merge is retry-safe. If the target merge succeeds but the apply-state update is interrupted, the next run sees the same batch again and ignores events whose _cedar_sequence is not newer.

​
Result table

The target table contains one current row per product_key. It includes the selected product fields plus Cedar metadata:
  • _cedar_sequence — the latest applied change sequence for that key;
  • _cedar_operation — upsert for current rows;
  • _cedar_commit_time — source commit time in epoch milliseconds;
  • _cedar_schema_version — pinned product contract version;
  • _cedar_snapshot_generation — subscription generation;
  • _cedar_deleted — current rows are false; rows delivered as true are removed; and
  • payload_json — the original Cedar product envelope.

​
Generation changes

A generation pins its schema and selected columns. If you create a new subscription or generation, use a new target table or complete a controlled replacement. Do not point an existing apply-state row at a different generation.

​
Troubleshooting

  • Snapshot manifest not found: Wait until Data Depot shows Snapshot delivered.
  • Permission denied: Configure Databricks read access to the landing zone. The credential entered in Data Depot is stored for Cedar delivery and is not returned.
  • Target already exists without Cedar apply state: Retry first. The notebook recovers the state when the table exactly matches the committed snapshot. If it reports a mismatch, choose a new table or verify and remove the incomplete target.

Related topics

Create Truck Load TenderUpdate Truck Load TenderGet Truck Load TenderList Truck Load TendersAPI Introduction