> ## Documentation Index
> Fetch the complete documentation index at: https://docs.cedarai.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Load Data Depot deliveries into Databricks

> Apply Cedar Product CDC snapshots and change batches to a current-state Delta table.

Data Depot can deliver the **Locations** and **Waybills** products as Parquet to your Azure or Amazon S3 landing zone. The Cedar Databricks notebook turns one delivery generation into a current-state Delta table in your workspace.

<Info>
  CDC delivery is currently an EU pilot for carriers on `eu-north-1:arms-prod`. Postgres JDBC and public API access
  remain available independently for other entitled carriers.
</Info>

<Warning>
  Do not point Auto Loader directly at the Cedar Parquet folders. Cedar uploads each Parquet object before its
  `manifest.json`. The manifest is the commit signal. Reading the Parquet path directly can process an incomplete or
  abandoned batch.
</Warning>

## Before you start

You need:

* a Databricks workspace with Delta Lake and access to create tables;
* a Unity Catalog external location, volume, or other Databricks storage configuration that can read your landing zone;
* a live Data Depot CDC delivery; and
* separate read access for Databricks. Data Depot stores the credential Cedar uses to write the delivery, but does not share that secret with Databricks.

Data Depot currently supports CDC delivery for Locations and Waybills. Other Postgres provisions are pull-only and do not produce this Parquet layout.

The initial snapshot is currently delivered as one Parquet object. Contact Cedar before onboarding a very large
carrier dataset so the first snapshot can be checked against the current delivery-worker capacity.

## 1. Create and verify the delivery

<Steps>
  <Step title="Create a subscription">
    In Data Depot, open **Delivery**, select **Subscribe to CDC**, then choose Locations or Waybills and an Azure or S3 destination.
  </Step>

  <Step title="Wait for the snapshot">
    Open the subscription and wait until **Snapshot delivered** has a timestamp. A `live` subscription can also have newer change batches.
  </Step>

  <Step title="Record the generation details">
    Copy the **Databricks base path** shown under **Databricks ingestion**. Data Depot converts Azure container URLs to
    `abfss://` and includes the destination prefix, subscription ID, and generation. The same panel provides the
    notebook download and this guide.
  </Step>
</Steps>

The underlying relative generation path has this form:

```text theme={null}
cedar-cdc/v1/subscription=<subscription-id>/generation=<generation>
```

Data Depot assembles it with your destination and optional prefix:

<Tabs>
  <Tab title="Amazon S3">
    ```text theme={null}
    s3://<bucket>/<optional-prefix>/cedar-cdc/v1/subscription=<id>/generation=<generation>
    ```
  </Tab>

  <Tab title="Azure Data Lake Storage">
    ```text theme={null}
    abfss://<container>@<account>.dfs.core.windows.net/<optional-prefix>/cedar-cdc/v1/subscription=<id>/generation=<generation>
    ```
  </Tab>
</Tabs>

## 2. Import the notebook

On the subscription detail page, select **Download notebook**. You can also download it from **Databricks** in Data
Depot. In your Databricks workspace, import `cedar_product_cdc_to_delta.py` as a source file.

The notebook has four parameters:

| Parameter      | Example                                                          | Purpose                                                  |
| -------------- | ---------------------------------------------------------------- | -------------------------------------------------------- |
| `base_path`    | `s3://customer-landing/cedar-cdc/v1/subscription=…/generation=7` | Full readable path to one subscription generation        |
| `generation`   | `7`                                                              | Generation shown in Data Depot                           |
| `target_table` | `main.cedar.waybills_current`                                    | New Delta table containing current rows                  |
| `state_table`  | `main.cedar.cdc_apply_state`                                     | Delta table storing the applied sequence for each target |

Use a new `target_table` when you load a generation for the first time. Multiple Cedar targets can share one `state_table`.

## 3. Run and schedule it

Run the notebook once interactively. It will:

1. require the committed snapshot manifest;
2. validate and load the snapshot;
3. expand the product fields from `payload_json` into columns;
4. find committed change manifests;
5. ignore already-applied rows from overlapping delivery ranges;
6. keep the newest event for each product key in a batch; and
7. update, insert, or delete rows with a sequence-guarded Delta `MERGE`.

After the first successful run, create a Databricks job using the same parameters. Schedule it at or after the delivery interval selected in Data Depot.

The merge is retry-safe. If the target merge succeeds but the apply-state update is interrupted, the next run sees the same batch again and ignores events whose `_cedar_sequence` is not newer.

## Result table

The target table contains one current row per `product_key`. It includes the selected product fields plus Cedar metadata:

* `_cedar_sequence` — the latest applied change sequence for that key;
* `_cedar_operation` — `upsert` for current rows;
* `_cedar_commit_time` — source commit time in epoch milliseconds;
* `_cedar_schema_version` — pinned product contract version;
* `_cedar_snapshot_generation` — subscription generation;
* `_cedar_deleted` — current rows are `false`; rows delivered as `true` are removed; and
* `payload_json` — the original Cedar product envelope.

## Generation changes

A generation pins its schema and selected columns. If you create a new subscription or generation, use a new target table or complete a controlled replacement. Do not point an existing apply-state row at a different generation.

## Troubleshooting

* **Snapshot manifest not found:** Wait until Data Depot shows **Snapshot delivered**.
* **Permission denied:** Configure Databricks read access to the landing zone. The credential entered in Data Depot is stored for Cedar delivery and is not returned.
* **Target already exists without Cedar apply state:** Retry first. The notebook recovers the state when the table exactly matches the committed snapshot. If it reports a mismatch, choose a new table or verify and remove the incomplete target.

## Related pages

* [Cedar API documentation](/user-docs/api-reference/introduction)
* [Identity and access management](/user-docs/iam/overview)


## Related topics

- [Create Truck Load Tender](/api-reference/truckloadtenderservice/create-truck-load-tender.md)
- [Update Truck Load Tender](/api-reference/truckloadtenderservice/update-truck-load-tender.md)
- [Get Truck Load Tender](/api-reference/truckloadtenderservice/get-truck-load-tender.md)
- [List Truck Load Tenders](/api-reference/truckloadtenderservice/list-truck-load-tenders.md)
- [API Introduction](/user-docs/api-reference/introduction.md)