How We Achieved 100x Cheaper Archive Storage Saving Millions
Cost reduction strategy for log storage using AWS S3 and Parquet.
1Cost Crisis and Solution Quest
In 2024, petabytes of MySQL RDS storage costs were skyrocketing. Databases nearing 64TB limits became critical, especially for 'cell history' and 'action log' tables with 10-year retention commitments. We needed an economical solution that matched MySQL's query performance for this cold data.
2Data Characteristics Analysis
90% of trillions of archive rows were rarely accessed cold data. Queries were typically point selections or paginated range queries filtered by specific 'base'. The data structure was ordered by MySQL's autoincr_id and largely read-only.
3Parquet Revolution
Parquet files in AWS S3 used 90% less space with columnar structure. Data partitioned by bases prevented irrelevant scans. Maintaining autoincr_id ordering enabled query engines to selectively download relevant byte ranges.
4Metadata Management
S3 locations of Parquet files were registered in DynamoDB. This custom metadata layer enabled customer-specific data residency and encryption. File metadata with min/max statistics and bloom filters minimized unnecessary data scans.
5Query Engine Selection
AWS Athena, DuckDB and DataFusion were tested. Athena's high latency and lack of isolation, plus DuckDB's suboptimal projection pushdowns led to choosing DataFusion. It delivered user-friendly subsecond latency consistently.
6Two-Tier Architecture
The system combined MySQL 'hot tier' for new data with S3 Parquet 'cold tier'. This hybrid approach preserved cost benefits without compromising user experience. Data integrity was prioritized during migration.
7Compression Success
Columnar Parquet format reduced dataset size to 1/10th of MySQL. Combined with S3's 10x cheaper storage, total costs dropped 100x - translating to millions in annual savings.
8Performance Optimizations
Smart organization of row groups and column chunks prevented unnecessary scans. Range queries on autoincr_id only downloaded relevant row groups from S3, reducing latency significantly.
9Migration Process
A custom pipeline enabled incremental migration without downtime. Old data converted to Parquet in batches while new data kept writing to MySQL. Data integrity checks were implemented at every stage.
10Security Measures
DynamoDB metadata layer addressed geographical data residency requirements. Customer-specific encryption keys ensured security. Granular S3 access controls were managed via AWS IAM policies.
11Scalability
The system was designed to handle petabytes seamlessly. Parquet partitioning strategy allowed adding new bases as data grew. S3's near-limitless scalability future-proofed the solution.
12Enterprise Impact
This innovation set new industry standards for archive storage costs. 10-year data retention commitments became financially sustainable. Architectural decisions established new cost-performance benchmarks.
Original article: Airtable Engineering — 100 Kat Daha Ucuz Arşiv Depolama ile Milyonlarca Tasarruf