Rebuilding GitHub Star History Without Exposing User Identities
GitHub optimizes API performance by replacing identity-heavy endpoints with efficient, aggregate star count data.
1The Challenge of Star History
Maintainers use star history to track project growth after releases or events. Previously, tools reconstructed these curves by requesting paginated lists of individual stargazers from the GitHub API. This design forced tools to discard user identities while keeping timestamps, creating significant inefficiencies and privacy risks for the platform and its users.
2Security and Privacy Risks
Publicly accessible stargazer lists became targets for bad actors scraping data for spam and phishing campaigns. To mitigate these risks, GitHub restricted access to stargazer and watcher lists, limiting visibility to repository administrators and collaborators. This security measure effectively cut off third-party tools that relied on the old API response structure.
3Inefficiency of Original Endpoints
The original stargazer endpoint was designed to identify who starred a repository, not to serve as a historical event log. For popular projects like react/react, which has 251,000 stars, generating a full curve required over 2,500 individual requests. This process frequently triggered rate limits and caused severe database performance bottlenecks.
4Database Performance Bottlenecks
A pagination bug forced the database to count all stars for every request, even when returning only 100 entries. On highly starred repositories, this resulted in thousands of expensive count queries. In the week before access was restricted, over 1,000 queries were killed by database safeguards, causing frequent request timeouts for users.
5Quantifying the Database Load
The scale of the performance issue was substantial, with nearly 37,000 slow count queries logged in a single week. Each of these requests averaged nearly three seconds of processing time. After restricting access to these lists, the database saw a 60% reduction in stargazer requests and an 80% decrease in total query time.
6Collaborative Solution Design
Following the API access changes, GitHub’s Community team engaged directly with tool builders like Star History. By understanding that maintainers needed growth curves rather than individual usernames, the product engineering team collaborated to design a purpose-built endpoint. This shift transformed a disruptive situation into a constructive partnership for better platform utility.
7New Aggregate API Endpoint
The new endpoint provides a direct way to retrieve star history by returning up to 30 weeks of daily star counts in a single request. By removing identity data from the response, the system no longer needs to process individual user records, significantly reducing the computational load on the underlying database infrastructure.
8Optimizing Data Processing
To keep the new endpoint efficient, GitHub implemented filtering by time before performing any counting operations. This approach ensures that the database only processes relevant data points, rather than scanning the entire history of a repository. This architectural change directly addresses the performance issues that plagued the previous, identity-based retrieval method.
9Focusing on Data Utility
Maintainers primarily use star history as an approximate signal of interest to compare activity before and after specific releases. The new design recognizes that tools only require counts and dates to visualize these trends. By decoupling identity from the API contract, GitHub successfully provided the necessary resolution without compromising user privacy.
10Impact on Tool Builders
The transition to an aggregate endpoint allows tool builders to maintain their functionality without needing access to sensitive user lists. By providing a dedicated path for historical data, GitHub enables developers to monitor project trajectories reliably. This change ensures that essential community tools remain operational while adhering to modern security standards.
11Evolution of API Design
This project highlights the importance of aligning API design with specific use cases. By moving from a generic 'who starred' query to a specialized 'historical count' query, GitHub improved both performance and security. This evolution demonstrates how platform providers can support ecosystem tools while simultaneously hardening their own internal database systems.
12Lessons in Platform Scaling
The successful reconstruction of star history serves as a case study in balancing community needs with platform stability. By listening to maintainers and re-engineering the underlying data model, GitHub resolved a critical performance issue. This outcome proves that collaborative problem-solving between platform teams and external developers leads to more sustainable, efficient software.