# Jane Street's Datacenter Near-Disaster: A Cooling System Failure How a false smoke alarm nearly destroyed a firm's critical trading infrastructure. - **Source:** [Hacker News: Front Page](https://brifing.si/source/b07aa772-b1a0-4a76-aa18-719c6f9e2dde) - **Published:** 2026-11-11 - **Topic:** [technology](https://brifing.si/c/technology) - **Original article:** - **On Brifing:** https://brifing.si/s/7021671e-3d9e-49cd-a544-0dcbd65f70e7 ## The Datacenter Proximity Strategy Jane Street historically maintained its primary datacenter inside its New York office building at 1 New York Plaza. This layout aimed to protect intellectual property from external spies and ensure that automated trading systems could be physically disabled by pulling power cables if they malfunctioned, reflecting a unique, submarine-inspired risk management culture. ## The False Alarm Trigger During routine construction in a prep room adjacent to the datacenter, dust kicked up by contractors triggered the facility's smoke alarm. This event initiated a dangerous automated sequence: the system shut down the Computer Room Air Conditioner units to prevent smoke circulation, causing the room temperature to rise rapidly toward critical levels. ## The Sprinkler System Risk The datacenter ceiling featured heat-sensitive sprinkler heads designed to activate if temperatures became extreme. While the smoke alarm pressurized the pipes, the lack of actual fire meant the heads remained closed. However, the shutdown of cooling units created a race against time before the rising heat triggered a catastrophic water release. ## Emergency Manual Intervention Alerted by the Hong Kong office, a systems team member living nearby rushed to the office to intervene. He manually disconnected power cables from server cabinets to halt heat generation. By physically cutting power, he successfully stabilized the temperature at 143.6 degrees Fahrenheit, narrowly avoiding the threshold that would have triggered the sprinklers. ## Post-Incident Operational Review Following the incident, the firm conducted matter-of-fact, engineering-focused meetings to assess the failure. Despite the near-loss of hardware, the company decided to retain the existing sprinkler setup and office-based datacenter location. They prioritized tightening operational procedures and surveillance protocols over moving the infrastructure to a more traditional, remote colocation facility at that time. ## Developing Reliable Telemetry The incident highlighted a critical gap in monitoring: the team only realized the danger because of a lucky observation of rising server temperatures. To prevent future occurrences, the author developed a custom monitoring tool using OCaml. This system polls every Linux server's onboard temperature sensors every minute to calculate a reliable average. ## The Philosophy of Durability The new monitoring system avoids reliance on potentially faulty CRAC telemetry or dedicated room sensors. By querying the servers directly, the system ensures that 'unknown unknowns' do not lead to hardware destruction. This approach reflects a preference for simple, type-safe, and direct data collection over complex, potentially unreliable integrated facility management systems. ## Legacy of the Near-Miss This event serves as a case study in the risks of co-locating high-frequency trading infrastructure with standard office environments. While the firm eventually faced larger challenges like Hurricane Sandy, the datacenter cooling fiasco remains a defining moment for the systems team, reinforcing the value of manual override capabilities and robust, decentralized monitoring. ## Sources covering this story - https://michael.bacarella.com/2026/11/10/false-smoke-alarm-jane-street-datacenter/ — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._