Jev-align CLI: Calibrate AI Classifiers with Human Feedback
Sutro's tool improves Jev models through iterative human labeling
1Interactive model refinement
Jev-align is a Python CLI tool that helps improve AI classifier functions built with TypeSafe's Jev framework. It identifies uncertain predictions in your dataset and prompts you to label them manually. The system then uses GEPA (Generative Evaluation and Proposal Architecture) to refine the model based on your feedback. This creates a continuous improvement cycle where human judgment trains the AI system.
2Flexible labeling options
The tool supports multiple annotation formats including binary (True/False), multiclass (single label), multilabel (multiple tags), and scoring (ordered rubric). Users can configure these through command-line flags, with each annotation type serving different evaluation needs. For multiclass and multilabel tasks, users define custom class names and descriptions, allowing for domain-specific categorization schemes.
3Data handling capabilities
Jev-align works with common file formats like CSV, Parquet, and JSONL, either processing the entire dataset or sampling the first 1,000 rows by default. Users can select specific columns for analysis or concatenate all fields. The tool includes three ready-to-run examples to help users get started quickly with real-world classification scenarios.
4Multi-provider deployment
The system supports multiple AI backend providers including TypeSafe AI (default), Vercel AI Gateway, and Cloudflare Workers AI. Each provider requires specific API key configurations. The chosen provider gets saved with the AI Function, ensuring consistent runtime behavior. This flexibility lets teams deploy models according to their existing infrastructure preferences.
5Reflection model configuration
GEPA's reflection model operates separately from the main JEV model, with automatic detection for OpenAI, Anthropic, and Gemini models. Advanced users can configure custom models through LiteLLM compatibility, supporting various providers including local vLLM instances and other OpenAI-compatible endpoints. This allows for specialized reflection models tuned to specific domains.
6Production feedback integration
Developers can load trained AI Functions into applications with capture mode enabled, collecting real-world examples during normal operation. These captured examples can later be labeled and fed back into the training process. This creates a virtuous cycle where production usage continuously improves model accuracy without disrupting service.
7Developer workflow
The CLI offers intuitive navigation with keyboard controls (arrow keys, Enter, Space) for efficient labeling sessions. Users can revisit previous labels or skip rationale prompts as needed. Saved AI Functions can be resumed later, with the system maintaining state between sessions. The tool also provides comprehensive help documentation for all command-line options.
8Open ecosystem approach
While developed by Sutro, Jev-align maintains compatibility with TypeSafe's Jev framework without formal affiliation. The tool exemplifies the growing ecosystem of specialized AI development utilities that interoperate through open standards. Its focus on human-in-the-loop refinement addresses a critical need in practical AI deployment - maintaining model relevance through continuous calibration.
Original article: Hacker News: Show HN — Jev-align CLI: Calibrate AI Classifiers with Human Feedback