Cloudflare has introduced Transformers for Logpush, a new capability that lets teams filter, redact, and reshape log streams using SQL before the data is exported. Instead of shipping raw logs to a destination and cleaning them up afterwards, Transformers apply changes directly to log data before it reaches the destination. The feature is powered by Apache DataFusion and Apache Arrow, giving it a columnar, query-engine foundation for processing log records in flight.
Concrete changes in this release:
- SQL-based processing on log streams, letting you express filtering, redaction, and reshaping logic declaratively rather than in downstream code.
- Filtering to drop unwanted records so destinations only receive relevant events.
- Redaction to remove or mask sensitive fields before logs leave Cloudflare's pipeline.
- Reshaping to restructure log fields into the schema your storage or analytics tool expects.
- Built on Apache DataFusion and Apache Arrow, which handle the query execution and columnar data representation.
For developers, the practical shift is where transformation happens. Previously, teams would land raw logs in a destination such as object storage or a SIEM, then run ETL jobs to strip PII, discard noise, or rename fields. That approach carries cost: you pay to store and transfer data you never wanted, and sensitive values may briefly exist in systems that should never hold them. Moving Transformers upstream into Logpush addresses both concerns at once by treating the log pipeline as a place to enforce policy.
That makes redaction a pipeline-level guarantee rather than a cleanup step, which matters for teams with data-residency or privacy requirements. It also reduces volume before export, which can lower egress and storage costs, and standardises schema so downstream dashboards and queries need fewer adapters. If you already manage Logpush jobs, the migration path is incremental: define the SQL you want, attach it as a Transformer, and verify the output at your destination before rolling it out broadly.
Source: https://blog.cloudflare.com/transformers-on-logpush/