Use Cedar data in Databricks - Cedar.AI Documentation

You can use Cedar data from the Databricks workspace you already run. You don’t need a separate Cedar Databricks account: Cedar provides the data and the credentials, and Databricks does the compute. The Databricks page in Data Depot summarizes three ways to connect.

Connect Databricks page with Read the Databricks guide, What Cedar doesn't do, and the Postgres via JDBC, Cedar APIs from notebooks, and Parquet landing to Delta cards — The three Databricks paths in Data Depot, for an EU carrier

Path	How it works	Status
Postgres via JDBC	Databricks reads a Data Depot Postgres view directly	Available
Cedar APIs from notebooks	A notebook calls Cedar’s public APIs with an API client key	Available
Parquet landing to Delta	Cedar pushes CDC files to your storage, and the Cedar notebook merges them into a Delta table	EU pilot. US on request through Cedar sales

Each card shows a badge with your own status for the carrier selected in the sidebar, so you can see which paths you can set up:

Badge	What it means
Available	You can set this path up yourself
EU pilot	You can set up CDC delivery, which is a pilot for carriers on Cedar’s EU production environment
View only	You can see what exists, such as views or API clients, but your role can’t create them. The card names the role to ask for
Not available	The path isn’t open to you for this carrier. The card says why, for example a missing role, or that CDC delivery isn’t offered for the carrier yet
Needs a carrier	Select a carrier in the sidebar first

Read the Databricks guide at the top of the page opens this guide. The What Cedar doesn’t do panel next to it sums up the limits: Cedar never writes to your Delta tables or their logs, doesn’t host a Unity Catalog or run a Databricks workspace for you, doesn’t use Databricks Partner Connect, and doesn’t share its delivery credential with Databricks.

Query Postgres views over JDBC

Best when you want live data on demand and don’t need to keep a copy.

Provision a view

In Data Depot, open Postgres, provision the source you need, and create its database login.⁠‌‌‌‌‌‌‌‌‌‌‌⁠ See Postgres views.

Store the login in Databricks

Save the username and password in a Databricks secret scope rather than in the notebook.

Read the view

Use the host, port, database, and view from the credential. SSL is required.

df = (
    spark.read.format("jdbc")
    .option("url", "jdbc:postgresql://<host>:<port>/<database>?sslmode=require")
    .option("dbtable", "<view>")  # for example cedar_apps.data_depot_waybills_<carrier>_<id>
    .option("user", dbutils.secrets.get("cedar", "depot-user"))
    .option("password", dbutils.secrets.get("cedar", "depot-password"))
    .load()
)

Each read queries Cedar’s current data. To keep history or a local copy, write the result to your own Delta table on a schedule. If your workspace must reach Cedar over a private network, see Private network connectivity. For a copy that runs on your own SLA rather than Cedar’s Postgres service, use CDC delivery into Delta.

Call Cedar APIs from a notebook

Best when you need data that isn’t a Postgres source, or want the same API responses your other integrations use.

Create an API client

In Data Depot, open API and create a client with the endpoints the notebook needs. See API clients.

Store the key

Save the key in a Databricks secret scope.

Call the API

Send the key in x-arms-api-key and an assumed user’s email in x-arms-assume-user.

import requests

resp = requests.post(
    "https://api-lg.arms.cedarai.com/ims/equipment/inventory",
    params={"carrierId": "<carrier ID>"},
    headers={
        "x-arms-api-key": dbutils.secrets.get("cedar", "api-key"),
        "x-arms-assume-user": "integration-bot@example.com",
        "Content-Type": "application/json",
    },
    json={"pageSize": 20},
)
resp.raise_for_status()

Cedar’s API permissions still apply to every call. For interactive questions instead of code, you can also connect an AI assistant with MCP.

Load CDC deliveries into Delta

Best when you want a current-state copy of Locations or Waybills in your lakehouse that updates every few minutes. Data Depot delivers each product as Parquet to your Azure or Amazon S3 landing zone, and the Cedar Databricks notebook turns one delivery generation into a current-state Delta table in your workspace.

CDC delivery is currently an EU pilot for carriers on Cedar’s EU production environment. We’re happy to make it available in the US; contact Cedar sales to discuss. Postgres JDBC and API access remain available for every carrier with Data Depot access.

Do not point Auto Loader directly at the Cedar Parquet folders. Cedar uploads each Parquet object before its manifest.json. The manifest is the commit signal. Reading the Parquet path directly can process an incomplete or abandoned batch.

Before you start

You need:

a Databricks workspace with Delta Lake and access to create tables;
a Unity Catalog external location, volume, or other Databricks storage configuration that can read your landing zone;
a live Data Depot CDC delivery; and
separate read access for Databricks. Data Depot stores the credential Cedar uses to write the delivery, but does not share that secret with Databricks.

The initial snapshot is currently delivered as one Parquet object. Contact Cedar before onboarding a very large carrier dataset so the first snapshot can be checked against the current delivery capacity.

1. Create and verify the delivery

Create a subscription

In Data Depot, open Delivery, select Subscribe to CDC, then choose Locations or Waybills and an Azure or S3 destination.⁠‌‌‌‌‌‌‌‌‌‌‌⁠ See CDC delivery.

Wait for the snapshot

Open the subscription and wait until Snapshot delivered has a timestamp. A Live subscription can also have newer change batches.

Record the generation details

Copy the Databricks base path shown under Databricks ingestion. Data Depot converts Azure container URLs to abfss:// and includes the destination prefix, subscription ID, and generation. The same panel provides the notebook download and this guide.

Databricks ingestion panel with the base path, Copy base path, Download notebook, and Read setup guide — The Databricks ingestion panel on a CDC subscription

The underlying relative generation path has this form:

cedar-cdc/v1/subscription=<subscription-id>/generation=<generation>

Data Depot assembles it with your destination and optional prefix:

Amazon S3
Azure Data Lake Storage

s3://<bucket>/<optional-prefix>/cedar-cdc/v1/subscription=<id>/generation=<generation>

abfss://<container>@<account>.dfs.core.windows.net/<optional-prefix>/cedar-cdc/v1/subscription=<id>/generation=<generation>

2. Import the notebook

On the subscription’s page, select Download notebook. You can also download it from the Parquet landing to Delta card on the Databricks page in Data Depot. In your Databricks workspace, import cedar_product_cdc_to_delta.py as a source file. The notebook has four parameters:

Parameter	Example	Purpose
`base_path`	`s3://customer-landing/cedar-cdc/v1/subscription=…/generation=7`	Full readable path to one subscription generation
`generation`	`7`	Generation shown in Data Depot
`target_table`	`main.cedar.waybills_current`	New Delta table containing current rows
`state_table`	`main.cedar.cdc_apply_state`	Delta table storing the applied sequence for each target

Use a new target_table when you load a generation for the first time. Multiple Cedar targets can share one state_table.

3. Run and schedule it

Run the notebook once interactively. It will:

require the committed snapshot manifest;
validate and load the snapshot;
expand the product fields from payload_json into columns;
find committed change manifests;
ignore already-applied rows from overlapping delivery ranges;
keep the newest event for each product key in a batch; and
update, insert, or delete rows with a sequence-guarded Delta MERGE.

After the first successful run, create a Databricks job using the same parameters. Schedule it at or after the delivery interval selected in Data Depot. The merge is retry-safe. If the target merge succeeds but the apply-state update is interrupted, the next run sees the same batch again and ignores events whose _cedar_sequence is not newer.

Result table

The target table contains one current row per product_key. It includes the selected product fields plus Cedar metadata:

_cedar_sequence — the latest applied change sequence for that key;
_cedar_operation — upsert for current rows;
_cedar_commit_time — source commit time in epoch milliseconds;
_cedar_schema_version — pinned product contract version;
_cedar_snapshot_generation — subscription generation;
_cedar_deleted — current rows are false; rows delivered as true are removed; and
payload_json — the original Cedar product envelope.

Generation changes

A generation pins its schema and selected columns.⁠‌‌‌‌‌‌‌‌‌‌‌⁠ If you create a new subscription or generation, use a new target table or complete a controlled replacement. Do not point an existing apply-state row at a different generation.

Troubleshooting

Snapshot manifest not found: Wait until Data Depot shows Snapshot delivered.
Permission denied on the landing zone: Configure Databricks read access to the landing zone. The credential entered in Data Depot is stored for Cedar delivery and is not returned.
Target already exists without Cedar apply state: Retry first. The notebook recovers the state when the table exactly matches the committed snapshot. If it reports a mismatch, choose a new table or verify and remove the incomplete target.
JDBC connection fails: Check that SSL is required in the URL (sslmode=require), that the password hasn’t been reset, and that your workspace can reach the host and port in the credential.
API calls return 401: Check the x-arms-api-key and x-arms-assume-user headers. See API clients.

​Query Postgres views over JDBC

​Call Cedar APIs from a notebook

​Load CDC deliveries into Delta

​Before you start

​1. Create and verify the delivery

​2. Import the notebook

​3. Run and schedule it

​Result table

​Generation changes

​Troubleshooting

​Related pages

Query Postgres views over JDBC

Call Cedar APIs from a notebook

Load CDC deliveries into Delta

Before you start

1. Create and verify the delivery

2. Import the notebook

3. Run and schedule it

Result table

Generation changes

Troubleshooting

Related pages