Jane Street's Datacenter Near-Disaster: A Cooling System Failure
How a false smoke alarm nearly destroyed a firm's critical trading infrastructure.
1The Datacenter Proximity Strategy
Jane Street historically maintained its primary datacenter inside its New York office building at 1 New York Plaza. This layout aimed to protect intellectual property from external spies and ensure that automated trading systems could be physically disabled by pulling power cables if they malfunctioned, reflecting a unique, submarine-inspired risk management culture.
2The False Alarm Trigger
During routine construction in a prep room adjacent to the datacenter, dust kicked up by contractors triggered the facility's smoke alarm. This event initiated a dangerous automated sequence: the system shut down the Computer Room Air Conditioner units to prevent smoke circulation, causing the room temperature to rise rapidly toward critical levels.
3The Sprinkler System Risk
The datacenter ceiling featured heat-sensitive sprinkler heads designed to activate if temperatures became extreme. While the smoke alarm pressurized the pipes, the lack of actual fire meant the heads remained closed. However, the shutdown of cooling units created a race against time before the rising heat triggered a catastrophic water release.
4Emergency Manual Intervention
Alerted by the Hong Kong office, a systems team member living nearby rushed to the office to intervene. He manually disconnected power cables from server cabinets to halt heat generation. By physically cutting power, he successfully stabilized the temperature at 143.6 degrees Fahrenheit, narrowly avoiding the threshold that would have triggered the sprinklers.
5Post-Incident Operational Review
Following the incident, the firm conducted matter-of-fact, engineering-focused meetings to assess the failure. Despite the near-loss of hardware, the company decided to retain the existing sprinkler setup and office-based datacenter location. They prioritized tightening operational procedures and surveillance protocols over moving the infrastructure to a more traditional, remote colocation facility at that time.
6Developing Reliable Telemetry
The incident highlighted a critical gap in monitoring: the team only realized the danger because of a lucky observation of rising server temperatures. To prevent future occurrences, the author developed a custom monitoring tool using OCaml. This system polls every Linux server's onboard temperature sensors every minute to calculate a reliable average.
7The Philosophy of Durability
The new monitoring system avoids reliance on potentially faulty CRAC telemetry or dedicated room sensors. By querying the servers directly, the system ensures that 'unknown unknowns' do not lead to hardware destruction. This approach reflects a preference for simple, type-safe, and direct data collection over complex, potentially unreliable integrated facility management systems.
8Legacy of the Near-Miss
This event serves as a case study in the risks of co-locating high-frequency trading infrastructure with standard office environments. While the firm eventually faced larger challenges like Hurricane Sandy, the datacenter cooling fiasco remains a defining moment for the systems team, reinforcing the value of manual override capabilities and robust, decentralized monitoring.