pg-cdc is a new open-source tool from the Burnside Project that streams PostgreSQL Write-Ahead Logs (WAL) directly into Iceberg tables on AWS S3, registering entities in the AWS Glue Catalog and gating reads with AWS Lake Formation tags. It is designed for environments where producers and consumers must be strictly separated.
Key features of pg-cdc:
- One-way, immutable pipeline – The WAL stream is one-way; data is flushed to Parquet files, which are immutable. Agents physically cannot write back to the source database.
- No database credentials for consumers – Consumers authenticate via AWS IAM and Lake Formation, never using a connection string. The tool itself requires only minimal database permissions to read the WAL.
- Governed by default – Lake Formation tags gate every read, down to the column. Untagged data is invisible. No extra configuration is needed to enforce column-level security.
- Time travel built in – Every flush creates an Iceberg snapshot with CDC epochs and
raw@tags, enabling historical queries without database branching. - Single binary, no JVM – The tool compiles to a single Go binary with no external dependencies. This simplifies deployment and reduces operational overhead.
- Typed, immutable Iceberg tables – Changes are captured as structured, typed Parquet columns suitable for analytics.
This design matters for developers building data pipelines because it eliminates the security risk of exposing production database credentials to analytics tools or AI agents. Instead, consumers authenticate through AWS IAM and only see the data that Lake Formation tags allow. Because Iceberg snapshots are immutable, developers can run point-in-time queries without extra infrastructure. The no-JVM approach also reduces memory and CPU overhead compared to tools like Debezium or Kafka Connect. For teams already using AWS, pg-cdc offers a simple, governed way to make Postgres data available for analytics while keeping production secure.