Have Cedar push a snapshot and ongoing change batches of Locations or Waybills as Parquet files to Azure or Amazon S3.
CDC delivery (change data capture) keeps a copy of a Cedar data product in your own cloud storage. Cedar first writes a
full snapshot, then pushes the changes as Parquet files on a schedule you choose: every 5, 10, or 15 minutes. You load
the files into your data lake or lakehouse, for example with the Cedar Databricks notebook.Choose CDC delivery when you want full control of the data. The data lands in storage you own, and you decide its
retention and who can access it. Your reports then run on your own SLA, not on Cedar’s Postgres service. See
Postgres views or CDC delivery? for a comparison.
CDC delivery is an EU pilot. Today you can subscribe only for carriers on Cedar’s EU production environment. For
other carriers, the Delivery page explains this, keeps existing deliveries visible, and turns off Subscribe to
CDC.Want CDC delivery in the US? We’re happy to make it available. Contact Cedar sales to discuss your requirements.
Meanwhile, Postgres views and API clients are
available for every carrier.
The Delivery page with destination options and existing deliveries
Your carrier’s location hierarchy and location template attributes, including deleted locations
Waybills
Waybill records covering the fields in the ARMS All Waybills grid
Each subscription delivers one data product to one destination. Deliveries include the full default set of columns for
that product.What CDC delivery doesn’t do:
Cedar never writes to your Delta or Iceberg tables or their logs. It writes files; you load them.
Destination credentials are accepted once, when you set up or rotate them, and never shown again.
For live SQL queries instead of files, use Postgres views.
Snowflake data sharing is still configured in ARMS Settings.
Create a place for Cedar to write, and a credential that lets it write there. Cedar checks the credential when you
subscribe by writing, reading, and deleting a small test file under .cedar-validate/ inside your prefix.
Azure Blob / ADLS Gen2
Amazon S3
Create a container (or pick an existing one) and, if you like, a folder prefix for Cedar, such as cedar.
Create a container SAS token with an expiry date that grants read, add, create, write, delete, and list
(racwdl) permissions on the container.
Have the container’s HTTPS URL ready, for example https://demorailanalytics.blob.core.windows.net/cedar-landing.
Open Delivery in the sidebar and select Subscribe to CDC.
2
Name the delivery and pick the data
Enter a Name that’s unique for the carrier, choose the View (Locations or Waybills), and choose the
Delivery interval: every 5, 10, or 15 minutes.
3
Enter the destination
Choose Azure Blob / ADLS Gen2 and enter the Container HTTPS URL, an optional Base prefix, and the
Container SAS token. Or choose Amazon S3 and enter the Bucket, Region, an optional Prefix, the
Access key ID, and the Secret access key.
Subscribing Waybills to an Azure container
4
Create the subscription
Select Create subscription. Data Depot checks the destination, and Cedar starts writing the snapshot.
For a carrier’s first subscription, Data Depot may ask you to retry in a few minutes while Cedar prepares delivery.
Wait a few minutes and submit again.
If the destination check fails, the subscription is created but waits for a working credential. Fix the access in Azure
or AWS, then return to Delivery and select Rotate credential on that subscription to run the check again.
The version of the data product’s layout this subscription uses. It stays the same for the life of the subscription
Generation path
Where this subscription’s files are written, relative to your container or prefix
Snapshot delivered
When the first full snapshot finished. Load nothing until this has a time
Last delivered
When Cedar last delivered a change batch
Through sequence
The latest change included in delivered files. It goes up as changes arrive
Validated
When Cedar last confirmed it can write to the destination
The Destination section shows the container URL or bucket details. Secrets aren’t shown. The
Databricks ingestion section gives you the exact path to load from, and links to the notebook and its guide.
Copy the base path and download the Databricks notebook
Under Delivery interval, pick a new Push every value and select Save interval. Cedar keeps capturing changes
the whole time; the interval only controls how often files are pushed.
On the Delivery page, Rotate credential appears under a delivery when its credential expires within 30 days, or the
delivery shows action required or pending validation. The badge turns from Expires in N days to critical within 7 days, and to
Credential expired after the expiry date.
Replace an Azure SAS token
1
Create the new credential
Create a new SAS token, or a new AWS access key, with the same permissions as before.
2
Rotate
Select Rotate credential, paste the new Container SAS token (Azure) or the Access key ID and Secret
access key (S3), and select Rotate credential.
3
Let Cedar check it
Cedar checks the new credential before it resumes delivery. Remove the old credential in Azure or AWS once the
delivery is live again.
The first snapshot and each change batch come with a manifest.json, which Cedar writes after the Parquet files it
describes. A batch is complete only once its manifest exists.
Don’t point Auto Loader or another file watcher straight at the Parquet folders: you could read a batch before it’s
complete. Load by manifest, as the Cedar Databricks notebook does.
CDC delivery is an EU pilot, available today only for carriers on Cedar’s EU production environment. Check the
carrier selected in the sidebar. For a US carrier, contact Cedar sales; we’re happy to discuss making CDC delivery
available for you.
Destination check did not succeed
Check that the SAS token or access key hasn’t expired and has the permissions listed in
Prepare your destination, and that the URL, bucket, region, and prefix are right. Then
select Rotate credential on the subscription to check again.
The name is already used
Subscription names must be unique for each carrier. Pick a different name.
A delivery shows action required
The destination credential usually stopped working, for example because the SAS token expired. Rotate the credential.
