Jane Street's Datacenter Near-Disaster Due to False Smoke Alarm
A systems engineer recounts racing to manually shut down servers before sprinklers triggered.
1Proximity and Operational Risk
Jane Street historically maintained its datacenter inside its Manhattan office building to protect intellectual property and ensure physical control over trading systems. This setup allowed engineers to manually pull power cables during emergencies. While unconventional for large firms, this risk-averse culture was heavily influenced by partners with nuclear submarine operational experience.
2The False Alarm Trigger
Contractors working in a prep room adjacent to the datacenter accidentally triggered a smoke alarm by generating dust. This event initiated a dangerous automated sequence within the building. The fire suppression system pressurized the sprinkler pipes, while the smoke detection logic simultaneously shut down the Computer Room Air Conditioner units.
3Escalating Thermal Threat
With cooling systems disabled, the datacenter temperature began to rise rapidly toward the 150 F threshold. If the heat-sensitive plugs on the sprinkler heads had melted, water would have flooded the server cabinets. The firm faced a potential total loss of its primary trading infrastructure due to this cascading failure.
4Emergency Manual Intervention
A systems engineer living nine minutes away received an urgent alert from the Hong Kong office. He sprinted to the 33rd floor of 1 New York Plaza to manually disconnect power cables from server racks. He successfully cut power as the room temperature reached 143.6 F, narrowly preventing the sprinkler activation.
5Post-Incident Technical Review
Following the near-catastrophe, the team held level-headed, engineering-focused meetings to evaluate procedures. They opted to retain the existing sprinkler setup while tightening surveillance protocols. The firm chose to keep servers in the office building, prioritizing immediate physical access over moving to a remote, third-party colocation facility at that time.
6Developing Reliable Telemetry
The incident highlighted the unreliability of existing cooling telemetry. To prevent future surprises, the engineer developed a custom monitoring tool using OCaml. This system queries every Linux server for its onboard temperature every minute. By calculating a fleet-wide mean, the team created a robust, fail-safe alert mechanism for generalized cooling failures.
7The Role of OCaml
The engineer chose OCaml for the new monitoring tool due to its type-safe nature and reliability. Unlike Bash or Perl scripts, this language provided the necessary stability for critical infrastructure monitoring. This approach reflects Jane Street’s broader engineering culture, which favors functional programming languages for mission-critical tasks across the entire firm.
8Legacy of Operational Grit
This event serves as a case study in the limitations of automated safety systems in dense urban office environments. While the firm eventually faced larger challenges like Hurricane Sandy, the datacenter incident reinforced the value of manual override capabilities and the necessity of building custom, reliable monitoring tools for unknown unknowns.
Original article: Hacker News — Jane Street's Datacenter Near-Disaster Due to False Smoke Alarm