Strot: Compile LLM Intent into Deterministic Local Decision Models
Eliminate runtime LLM costs by compiling intent into frozen, type-safe local artifacts.
1Shift LLM Logic to Compile-Time
Strot introduces a paradigm shift by moving LLM inference from runtime to compile-time. Developers define intent and schemas once, which the system uses to synthesize a frozen, local Python artifact. This approach eliminates the need for live LLM calls during production, ensuring that every record is processed with zero marginal cost.
2Deterministic Decision Execution
At runtime, Strot functions as a lightweight, local decision model rather than a generative agent. It processes records using a 144M parameter model, such as the default Julia-1, which operates entirely on local CPU hardware. This architecture ensures that outputs are bounded, type-safe, and free from the structural hallucinations typical of LLMs.
3Confidence-Based Abstention Mechanism
The system incorporates split conformal prediction and temperature scaling to provide statistical guarantees on decision accuracy. If the model encounters a record where confidence falls below a set threshold, it abstains from guessing. This mechanism ensures that dispatched decisions maintain high reliability, as the system only commits to outputs that pass rigorous calibration.
4Schema-Driven Workflow Integration
Developers interact with the framework through two primary surfaces: run() for routing records to specific handlers, and decide() for answering typed questions. By defining an action registry and specific instructions, users can perform complex tasks like sentiment scoring or bug classification while receiving structured JSON outputs containing probabilities and confidence scores.
5Golden Record Validation
Strot enforces quality through a golden-record promotion process. Before an artifact is finalized for production, it must pass a set of known answers provided by the developer. This ensures that the compiled decision logic aligns with expected outcomes, preventing regressions and maintaining high performance standards across the entire deployment lifecycle.
6Model-Agnostic Runtime Architecture
While Strot defaults to the SupersonicLabs Julia-1 model, the architecture is designed to be model-independent. The decision model acts as a pluggable slot, allowing developers to swap in hosted APIs or specialized local models as needed. This flexibility ensures the system can adapt to evolving requirements without changing the core compilation pipeline.
7Operational Efficiency and Deployment
Production deployment is simplified by baking the local model directly into the container image. By using the STROT_NLU_CHECKPOINT environment variable, teams can ensure consistent behavior across environments. This offline-first approach removes external dependencies, significantly reducing latency to approximately 60 milliseconds per record while maintaining full control over the underlying infrastructure.
8Future of Deterministic AI
By treating LLM intent as a source for deterministic code synthesis, Strot addresses the primary barriers to enterprise AI adoption: cost, latency, and unpredictability. This framework demonstrates that complex classification and routing tasks can be handled reliably at scale, provided the intelligence is distilled into specialized, verifiable, and local decision models.
Original article: Hacker News — Strot: Compile LLM Intent into Deterministic Local Decision Models