Automating Prompt Engineering with JetBrains' Koog Optimization Library
Optimize agent instructions automatically using feedback loops and iterative refinement in Kotlin.
1Automating Agent Instruction Tuning
JetBrains Research introduced the Koog Agents Optimization Library to automate prompt refinement for Kotlin-based agents. Instead of manual prompt engineering, developers define optimizable subgraphs within their agent architecture. The system runs the agent against a training set, evaluates performance, and generates improved instructions as a JSON artifact without requiring code rewrites.
2Integrating the Optimization Workflow
The library replaces standard subgraphs with optimizable versions, allowing developers to isolate specific instructions for tuning. By providing a scoring function and a feedback mechanism, the optimizer iterates through multiple candidates. This process keeps several instructions alive simultaneously, ensuring that the model learns from both failures and successes during training rounds.
3Benchmark Performance on PUPA
Testing on the PUPA privacy benchmark demonstrated significant gains in agent efficacy. The baseline score of 81.19 improved to 94.22 after the optimization process. These results, calculated as the mean of five distinct runs, confirm that the automated feedback loop effectively enhances agent performance while maintaining strict privacy-preserving data handling.
4Mechanism of Instruction Evolution
The optimizer transforms vague, two-line instructions into highly detailed, multi-character directives. For instance, the system autonomously generated specific rules for redaction, such as replacing professor names and course codes with neutral placeholders. This evolution occurs entirely through iterative testing, uncovering optimal strategies that developers might not have explicitly defined themselves.
5Balancing Privacy and Utility
The generated instructions reflect a complex trade-off between zero data leakage and task utility. The system learned to preserve enough context for the LLM to execute the user's request accurately while ensuring no private information is exposed. This emergent strategy demonstrates the power of automated tuning in handling sensitive data pipelines.
6Standardized Optimizer Interface
The library provides a consistent interface across different optimization algorithms, including GEPA, BootstrapFewShot, MIPROv2, and ACE. Developers can switch between these methods by changing a single constructor, as the train and loadOptimizedAgent methods remain uniform. This modular design allows for easy experimentation with various strategies without altering the core agent code.
7Cost-Effective Optimization Options
For teams seeking efficiency, BootstrapFewShot serves as a cost-effective starting point by mining demonstrations from successful runs without additional LLM calls. Conversely, ACE allows agents to accumulate knowledge through a playbook of insights rather than just refining prompts. These diverse options cater to different performance requirements and computational budget constraints.
8Future of Agentic Development
This research highlights a shift toward treating agent instructions as dynamic artifacts rather than static strings. By integrating optimization directly into the Kotlin development lifecycle, JetBrains provides a scalable path for improving complex agent behaviors. Future updates are expected to merge the optimizable-subgraph support into the main Koog repository for broader adoption.