Inside GitHub’s New Git Storage Architecture
GitHub unveils Rubicon, a scalable storage architecture designed to support massive agentic development workloads.
1New Storage Architecture
GitHub recently introduced a new Git storage architecture designed to handle thousands of writes per second. This system supports 5,000 concurrent agents working on a single repository while maintaining low-latency read performance. The infrastructure is built to scale independently, ensuring that GitHub can support the growing demands of automated agentic development.
2Core Design Principles
The architecture relies on two fundamental principles: minimizing coordination and decoupling storage from compute. By separating these concerns, GitHub allows different components to scale independently. This design shift ensures that Git operations remain familiar to developers while providing the backend robustness required for high-concurrency environments and massive repository traffic.
3The Rubicon Project
Launched in May 2026, Project Rubicon serves as the foundation for this new storage model. It utilizes Azure Blob Storage for durable data and a specialized service called Ref-CAS for reference updates. This approach allows GitHub to maximize throughput while maintaining strict correctness, durability, and availability guarantees for all users.
4Handling Horizontal Scale
GitHub maintains horizontal scale by partitioning tenants across infrastructure and enforcing traffic fairness. Spokes continues to handle routing and load balancing, ensuring that requests are distributed effectively. These battle-tested methods allow the platform to manage many developers and agents working across millions of repositories without compromising overall system performance or stability.
5Vertical Scaling Challenges
Vertical scale focuses on how many agents can operate on a single repository simultaneously. As agentic workflows increase, GitHub expects sustained high-volume bursts rather than short-lived spikes. The new architecture targets two orders of magnitude more scale, ensuring the platform remains performant as individual repository sizes and activity levels grow exponentially.
6Virtual Replica Fleet
Spokes routes requests to virtual replicas, which are disposable compute nodes that serve reads and writes. These replicas use local disk caches to keep latency low. Because correctness does not depend on a specific replica, the system can dynamically add more nodes as repository activity increases, ensuring consistent performance during peak usage.
7Ref-CAS Service Role
Ref-CAS, or Reference Compare-and-Swap, is a replicated log service dedicated to managing reference state. Unlike legacy systems, it does not store or validate object data, only pointers. This allows the system to process tens of thousands of reference updates per second, providing a high-throughput baseline for repository versioning and synchronization.
8Optimizing Write Throughput
To achieve high throughput, GitHub moved away from using Blob Storage for reference state. While object storage is excellent for durable data, it struggles with high-frequency reference updates. By implementing a dedicated consensus-based log service, the team successfully eliminated the write bottlenecks that limited previous prototypes to only mid-hundreds of operations.
9Decoupled Compaction Nodes
In the new architecture, compaction runs entirely in the background on separate nodes. These nodes work directly against Blob Storage and Ref-CAS to merge packs and remove unreachable objects. This separation ensures that virtual replicas can dedicate their CPU and memory resources exclusively to serving incoming read and write requests.
10Read and Write Flow
When a write occurs, a virtual replica indexes the pack, uploads it to Blob Storage, and commits a transaction to Ref-CAS. For reads, the replica checks the version against Ref-CAS and pulls necessary packs into its cache. This workflow ensures that reads remain cheap and fast even when repository state evolves.
11System Fault Tolerance
Spreading write operations across multiple nodes improves fault tolerance and speeds up recovery during node failures. Because indexing and object connectivity checks are CPU-intensive, distributing this work prevents any single node from becoming compute-bound. This modular approach allows GitHub to maintain high availability even under extreme, sustained traffic from automated agents.
12Future of Development
This architecture represents a significant shift toward supporting the future of agentic development. By planning for exponential growth in traffic and read fan-out, GitHub ensures its infrastructure will not become a bottleneck. The transition to this decoupled, high-throughput model provides the necessary foundation for the next generation of software engineering tools.
Original article: GitHub Engineering — Inside GitHub’s New Git Storage Architecture