PostHog Migrates Data Warehouse Architecture Away From DuckDB
Transitioning to Trino and HogLake to overcome single-node scaling limitations at petabyte volume.
1Transitioning from DuckDB
PostHog is moving its data warehouse away from DuckDB after five months of operation. While DuckDB excels as a single-node analytical tool, it struggled to handle the complex, large-scale workloads of PostHog customers. The team prioritized a more predictable, scalable experience over the initial benefits of DuckDB's local-first architecture.
2Single-node Cloud Limitations
Operating a single-node engine in the cloud created significant resource management challenges. Predicting CPU and memory requirements for diverse customer queries proved difficult, leading to inefficient instance sizing. Furthermore, the lack of asynchronous fetching from S3 created bottlenecks that hindered performance when processing the massive datasets typical of PostHog's scale.
3DuckLake Scalability Hurdles
The DuckLake extension, while simple, lacked the distributed design required for PostHog's specific needs. Catalog schema portability across SQL databases limited engine-specific optimizations. Consequently, single-table writes were slowed by frequent re-reading of statistics, and metadata database connections became increasingly expensive to maintain on shared Postgres instances during high-volume operations.
4Managing Trillion-Row Datasets
PostHog dogfooded its own warehouse, reaching approximately 2 petabytes of data, 1 trillion rows, and 15.3 million snapshots. This massive scale exposed fundamental issues within the DuckLake scope that were difficult to patch. Maintaining a custom fork of the software ultimately negated the community-driven benefits of using the open-source tool.
5Postgres Compatibility Challenges
Offering direct Postgres connections via port 5432 proved more complex than anticipated. Managing long-running connections, idle timeouts, and load balancing for pgwire lacked the robust tooling available for HTTP. Additionally, the team spent excessive engineering hours addressing niche compatibility issues across a vast array of different Postgres client libraries.
6Shifting to Trino Architecture
The team is rebuilding the warehouse infrastructure using Trino, a distributed query engine designed for scale. This shift moves away from the single-node constraints of their previous setup. By focusing on a distributed architecture, PostHog aims to provide a more reliable and performant experience for users running large-scale analytical queries.
7Introducing HogLake Format
Alongside Trino, PostHog developed HogLake, a custom open table format tailored to their specific performance requirements. This internal development allows the team to implement optimizations that were previously impossible under the strict compatibility constraints of DuckLake. They are currently testing this new architecture at petabyte scale before wider implementation.
8Strategic Infrastructure Evolution
This migration highlights the trade-offs between simple, single-node tools and distributed systems at high data volumes. PostHog's move underscores that while local-first tools are brilliant for specific use cases, enterprise-grade warehouses require native distributed capabilities. The focus remains on delivering predictable performance and cost-efficiency for customers handling massive analytical workloads.