Have Cedar push a snapshot and ongoing change batches of Locations or Waybills as Parquet files to Azure or Amazon S3.
CDC delivery (change data capture) keeps a copy of a Cedar data product in your own cloud storage. Cedar first writes a
full snapshot, then pushes the changes as Parquet files on a schedule you choose: every 5, 10, or 15 minutes. You load
the files into your data lake or lakehouse, for example with the Cedar Databricks notebook.Choose CDC delivery when you want full control of the data. The data lands in storage you own, and you decide its
retention and who can access it. Your reports then run on your own SLA, not on Cedar’s Postgres service. See
Postgres views or CDC delivery? for a comparison.
CDC delivery is an EU pilot. Today you can subscribe only for carriers on Cedar’s EU production environment. For
other carriers, the Delivery page says CDC delivery isn’t available for this carrier yet, keeps existing
deliveries visible, and turns off Subscribe to CDC.Want CDC delivery in the US? We’re happy to make it available. Contact Cedar sales to discuss your requirements.
Meanwhile, Postgres views and API clients are
available for every carrier.
Your carrier’s location hierarchy and location template attributes, including deleted locations
Waybills
Waybill records covering the fields in the ARMS All Waybills grid
Each subscription delivers one data product to one destination. Deliveries include the full default set of columns for
that product.What Cedar doesn’t do:
Cedar never writes to your Delta or Iceberg tables or their logs. It writes files; you load them.
Destination credentials are accepted once, when you set up or rotate them, and never shown again.
For live SQL queries instead of files, use Postgres views.
Snowflake data sharing is still configured in ARMS Settings.
Create a place for Cedar to write, and a credential that lets it write there. Cedar checks the credential when you
subscribe by writing, reading, and deleting a small test file under .cedar-validate/ inside your prefix. The
Prepare your destination cards on the Delivery page list the same steps.
Azure Blob / ADLS Gen2
Amazon S3
Create a container (or pick an existing one) and, if you like, a folder prefix for Cedar, such as cedar.
Create a container SAS token with an expiry date and read, add, create, write, delete, and list (racwdl)
permissions on the container prefix. Data Depot doesn’t accept a SAS token without an expiry date.
Have the container’s HTTPS URL ready, for example https://demorailanalytics.blob.core.windows.net/cedar-landing.
Open Delivery in the sidebar and select Subscribe to CDC. Or select Subscribe with Azure or Subscribe
with Amazon S3 on the Prepare your destination cards to start with that storage chosen.
2
Name the delivery and pick the data
Choose the View (Locations or Waybills) and the Delivery interval: every 5, 10, or 15 minutes. Data Depot
suggests a Name, such as “Waybills to Azure”; change it if you like. The name must be unique for the carrier.
3
Enter the destination
Under Storage, choose Azure Blob / ADLS Gen2 and enter the Container HTTPS URL, an optional Base
prefix, and the Container SAS token. Or choose Amazon S3 and enter the Bucket, Region, an optional
Prefix, the Access key ID, and the Secret access key.
Subscribing Waybills to an Azure container
4
Create the subscription
Select Create subscription. Data Depot checks the destination and opens the new subscription’s page, where you
can follow Cedar writing the snapshot.
For a carrier’s first subscription, Data Depot may say Cedar is preparing delivery for this carrier. Wait a few
minutes, then select Create subscription again.
If the destination check fails, the subscription is still created, and waits for a working credential. Data Depot shows
Destination check did not succeed with the reason and what the credential needs. Fix the access in Azure or AWS,
then select Rotate credential right there to enter the credential again and rerun the check. You can also select
View subscription and rotate it later.
The CDC deliveries list at the top of the Delivery page shows each subscription with its storage, data product,
interval, credential expiry, and status. When a delivery needs a new credential, Rotate credential is on the right
of its row, and a delivery that stopped shows the reason underneath.
CDC deliveries: one expiring soon, one with an AWS access key, and one that needs a new credential
Status
Meaning
Pending validation
Cedar is checking the destination, or is waiting for a working credential
Snapshotting
The destination works and Cedar is writing the first full snapshot
Waiting for route
Cedar is still preparing delivery for this carrier
Live
The snapshot is delivered and change batches are flowing on schedule
Action required
Delivery stopped and needs you, usually because the destination credential stopped working
Paused
Delivery is paused
Deleted
The subscription was deleted and Cedar no longer delivers
Select a delivery to open its page. The heading is the delivery’s name, with its status and a sentence that explains
what the status means. When the credential needs attention, Rotate credential is at the top right.
A live delivery's page
Field
What it tells you
Generation
The version of the data product’s layout this subscription uses. It stays the same for the life of the subscription
Generation path
Where this subscription’s files are written, relative to your container or prefix
Snapshot delivered
When the first full snapshot finished. Load nothing until this has a time
Last delivered
When Cedar last delivered a change batch
Through sequence
The latest change included in delivered files. It goes up as changes arrive
Validated
When Cedar last confirmed it can write to the destination
The Destination section shows the storage, the container URL or bucket details, and when the credential expires.
Secrets aren’t shown. The Databricks ingestion section gives you the exact path to load from, and links to the
notebook and its guide.
Copy the base path and download the Databricks notebook
Under Delivery interval, pick a new Push every value (5, 10, or 15 minutes) and select Save interval. Cedar
keeps capturing changes the whole time; the interval only controls how often files are pushed.
Rotate credential appears on the right of a delivery in the list, and at the top right of its page, when its
credential expires within 30 days, or the delivery shows Action required or Pending validation. An Expires in
N days badge appears 30 days before the expiry date and turns red in the last 7 days. After the date, it reads
Credential expired.
Replace an Azure SAS token
1
Create the new credential
Create a new SAS token, or a new AWS access key, with the same permissions as before.
2
Rotate
Select Rotate credential, paste the new Container SAS token (Azure) or the Access key ID and Secret
access key (S3), and select Rotate credential.
3
Let Cedar check it
Cedar checks the new credential before it resumes delivery. Remove the old credential in Azure or AWS once the
delivery is Live again.
The first snapshot and each change batch come with a manifest.json, which Cedar writes after the Parquet files it
describes. A batch is complete only once its manifest exists.
Don’t point Auto Loader or another file watcher straight at the Parquet folders: you could read a batch before it’s
complete. Load by manifest, as the Cedar Databricks notebook does.
CDC delivery is an EU pilot, available today only for carriers on Cedar’s EU production environment. Check the
carrier selected in the sidebar. For a US carrier, contact Cedar sales; we’re happy to discuss making CDC delivery
available for you. On an EU carrier, your role may let you see deliveries but not create or change them. The
Delivery page then says so at the top and names the role to ask for.
Destination check did not succeed
Check that the SAS token or access key hasn’t expired and has the permissions listed in
Prepare your destination, and that the URL, bucket, region, and prefix are right. Then
select Rotate credential on the subscription to check again.
The name is already used
Subscription names must be unique for each carrier. Pick a different name.
A delivery shows Action required
The destination credential usually stopped working, for example because the SAS token expired. Rotate the credential.
