Inside GitHub’s New Git Storage Architecture
GitHub unveils Project Rubicon to support massive agentic development workloads with high-throughput storage.
1Scaling for Agentic Development
GitHub recently unveiled a new Git storage architecture designed to handle thousands of writes per second. This system supports 5,000 concurrent agents working on a single repository. By enabling 7,200 pushes per second, the platform prepares for a future where automated agents perform continuous code development and deployment tasks at scale.
2Core Design Principles
The architecture relies on two fundamental principles: minimizing coordination and decoupling storage from compute. By separating these concerns, GitHub allows different infrastructure components to scale independently. This design ensures that Git remains familiar to developers while providing the performance required for high-frequency, automated, and parallelized code repository interactions.
3The Role of Virtual Replicas
Virtual replicas serve as a disposable compute fleet that handles both read and write traffic. These nodes maintain local caches that can be rebuilt at any time, ensuring high availability. When a repository experiences increased activity, GitHub dynamically adds more virtual replicas to maintain low latency and consistent performance for users.
4Project Rubicon Overview
Project Rubicon represents the transition to this new storage model, initiated in May 2026. It utilizes Azure Blob Storage for durable data and a specialized service called Ref-CAS for reference updates. This structure ensures that Git objects remain immutable while reference updates are handled efficiently through a high-throughput consensus-based log.
5Ref-CAS Service Mechanism
Ref-CAS, or Reference Compare-and-Swap, is a dedicated replicated log service for managing repository references. It allows multiple writers to update references in parallel without contention. Each transaction is committed by a quorum, ensuring consistency while enabling tens of thousands of operations per second, far exceeding previous limitations of blob-based storage.
6Azure Blob Storage Integration
Nearly all durable data, including packs, indexes, and read accelerators, resides in Azure Blob Storage. This service provides high aggregate throughput and cost-effective storage for immutable Git objects. By offloading data storage to Azure, GitHub ensures that its compute nodes can focus entirely on serving requests and maintaining system performance.
7Handling Write Throughput
The architecture supports multiple writer nodes to prevent compute bottlenecks during indexing and connectivity checks. Spreading these CPU-intensive tasks across many nodes improves fault tolerance and accelerates recovery times. This horizontal distribution is essential for maintaining commit latency targets as agent-driven development continues to increase the volume of repository pushes.
8Optimizing Read Performance
Reads remain inexpensive because they only require a version check against the Ref-CAS log. If a virtual replica is current, the request incurs only a single comparison cost. If the replica is behind, it fetches only the necessary data pointers, minimizing unnecessary data transfer and keeping read traffic highly efficient.
9Background Compaction Nodes
Compaction nodes operate independently of virtual replicas, working directly against Blob Storage and Ref-CAS. These nodes merge packs and remove unreachable objects in the background, ensuring that read performance does not degrade over time. By offloading this work, virtual replicas dedicate their CPU and memory resources exclusively to serving user requests.
10Horizontal and Vertical Scaling
GitHub addresses scale across two axes: horizontal scaling for many repositories and vertical scaling for busy individual repositories. While horizontal scaling uses established partitioning and load-aware routing, vertical scaling required this new architecture. The system is built to handle exponential traffic growth, targeting two orders of magnitude more scale than before.
11Maintaining Git Compatibility
Despite the significant backend changes, the developer experience remains unchanged. Users continue to push, fetch, and branch using standard Git commands. The platform maintains all existing protections and workflows, ensuring that the transition to the new storage architecture is transparent to the millions of developers relying on GitHub daily.
12Future of Repository Traffic
GitHub anticipates that agent-driven development will transform repository traffic from short bursts into sustained, high-volume streams. The new architecture provides the necessary foundation to support this shift. By planning for exponential growth, GitHub ensures its infrastructure remains robust and capable of supporting the next generation of automated software development tools.
Original article: The GitHub Blog — Inside GitHub’s New Git Storage Architecture