Counter Service Storage Migration: Design Choices and Learnings
Performance and security strategies in high-volume data storage migration.
1Introduction: The Critical Role of Counter Service
Counter Service in Grab's anti-fraud platform handles time-windowed queries like recent ride requests per user or failed payment attempts, processing about a billion requests daily. The service maintains strict latency and reliability requirements to support real-time fraud rule evaluation.
2Challenges of Existing Infrastructure
The wide-column database system began showing performance issues as it scaled. Each counter event triggered three parallel SELECT queries and a batch write operation, resulting in four network round-trips per event and causing high latency.
3Decision to Migrate to Aerospike
An organizational infrastructure review identified Aerospike as better suited for Counter Service. The migration presented an opportunity to decouple storage concerns from business logic and reduce future change efforts.
4Data Model Re-evaluation
During migration, the data model and access patterns were thoroughly reviewed. A new optimized structure was created for data stored in 15-minute, hourly, and daily buckets, significantly improving query performance.
5Safe Migration Strategy
Due to high query volume, uninterrupted and secure migration was critical. Three key requirements were set: gradual traffic ramp-up, data integrity monitoring, and zero-downtime migration. This allowed rollback at any point via configuration changes.
6Reader Service Restructuring
The Rust-based reader service's tightly coupled business logic with storage was rearchitected. Code split into storage/legacy.rs and batch_read_ops.rs created a more modular structure, facilitating support for different storage backends.
7Storage Abstraction Layer
The new storage/ module with legacy.rs, aerospike.rs and mock_storage.rs files created a single access point. Enum-based routing allowed runtime backend selection without performance overhead.
8Writer Pipeline Updates
An optimized data model was designed for write operations, replacing read-modify-write cycles with a more efficient approach. This reduced network round-trips and improved overall system performance.
9Gradual Rollout Process
Migration was implemented in controlled phases. Both old and new storage paths were monitored at each stage, with full transition occurring after data integrity verification, maintaining continuous system operation.
10Performance Improvements
Post-migration to Aerospike, the system achieved lower latency and higher query capacity. The redesigned data model optimized resource usage, reducing costs while improving real-time fraud detection reliability.
11Future Flexibility
The changes not only facilitated the current migration but also created a framework for future storage changes. The abstraction layer significantly simplified integration of new technologies.
12Conclusion and Key Takeaways
This project demonstrated how to safely change storage infrastructure in high-volume critical systems. Decoupling business logic from storage, designing efficient data models, and phased rollout strategies were key to success, creating valuable reference for future infrastructure projects.
Original article: Grab Tech — Counter Service Depolama Geçişi: Tasarım Seçimleri ve Öğrenilenler