Cloudflare Embeds In-Process SQL Engine into Logpush Pipeline Using Apache DataFusion
Embedded Rust query engine enables SQL log filtering and redaction without external middleware.
1In-Stream Log Transformation
Cloudflare has made Transformers generally available across all self-serve and enterprise Logpush plans. The feature allows observability teams to execute SQL queries directly on log batches before export. This setup filters unwanted events, masks sensitive information, and standardizes field schemas without requiring separate intermediate processing middleware.
2Direct Downstream Cost Savings
Observability teams typically route CDN logs to SIEMs, object storage, and analytics platforms. Ingesting health checks or raw unformatted records inflates destination licensing and storage expenses. Moving filtering logic directly into the edge export stage reduces downstream data volumes and eliminates custom infrastructure maintenance for log cleanup.
3Standardized SQL and NDJSON
Transformers apply standard SQL syntax against specific dataset schemas like HTTP request logs. The query evaluates batch records, dropping entries that fail WHERE conditions and restructuring output via SELECT projections. Exported payloads are delivered as NDJSON, featuring normalized fields, redacted IPs, and structured JSON properties.
4High-Throughput Delivery Architecture
Cloudflare's ingestion pipeline processes over one billion events per second, while the Go-based Bufstream service exports 110 million records per second. Bufstream runs across Kubernetes pods consuming data from the internal Buftee message broker. Workloads follow a Pareto distribution, where select high-volume jobs handle millions of logs every second.
5Rejecting Out-of-Process Workers
Initial prototypes ran user-defined WebAssembly transformations inside Cloudflare Workers. However, invoking Workers introduced external network calls into the high-throughput export path, causing transient timeouts and latency spikes. Cloudflare chose an embedded in-process execution model to preserve predictable throughput, fault tolerance, and zero-network overhead during log delivery.
6DataFusion and Arrow Engine
Engineers embedded Apache DataFusion, a Rust-based vectorized SQL engine built on Apache Arrow, inside Bufstream. In synthetic benchmarks on 5,000-row HTTP request batches, single goroutine transform execution completed in 3 milliseconds, achieving 1.6 million rows per second. High-volume export jobs utilize per-job transform parallelism across pods.
7Go-Rust FFI Memory Safety
Bufstream converts decompressed Cap'n Proto records into typed Apache Arrow RecordBatches, passing data to Rust through Cgo and Arrow IPC. The Go and Rust runtimes avoid shared memory ownership: Go provides pointers during the Cgo call, copies the resulting Rust buffer, and explicitly frees Rust allocations to avoid leaks.
8Fail-Closed Execution Guarantees
Transformers enforce a fail-closed operational policy. If a transform query errors, Bufstream halts delivery offsets and avoids sending raw, unredacted records to destinations. The system progressively delays retries for failing jobs and eventually disables persistent failures to protect cluster capacity, providing health monitoring dashboards for customers.
9Serverless Pipeline Simplification
Embedding analytical execution engines directly within delivery pipelines marks a major architectural shift for streaming infrastructure. By executing vectorized SQL directly across memory boundaries, Cloudflare eliminates dedicated log extract-transform-load clusters. Teams gain granular data governance, lower transit costs, and simplified cross-cloud analytics pipelines.