Building a Jev-like wrapper for LLMs with vision capabilities
A Python implementation for real-time webcam analysis using multimodal models
1Jev-inspired LLM wrapper
The project implements a Jev-like wrapper for LLMs that leverages token probabilities for decision-making. By forcing models to output single tokens, it enables quick analysis while maintaining flexibility. The wrapper works with both text-based and vision-capable models, extending Jev's original concept to handle image inputs through a custom attachments field. This approach allows for rapid question-answering against visual inputs while maintaining the simplicity of the Jev format.
2Token probability technique
The system uses a clever trick with LLM token probabilities to make decisions. By structuring prompts to require single-letter responses and enabling logprobs in API calls, it captures the model's confidence in each option. This method avoids lengthy outputs while still providing nuanced results. The approach works with both local models (like Gemma through llama.cpp) and commercial APIs (like OpenAI), though implementation details differ between platforms. The technique is particularly effective for multiple-choice style questions against visual inputs.
3Vision model implementation
The wrapper extends Jev's capabilities to handle vision models by adding an attachments field for images. Webcam frames are captured, encoded as base64 JPEGs, and sent alongside text prompts. The example demonstrates analyzing scenes for person visibility, environment type, and lighting conditions. On an RTX 3090 with Gemma 4 12B, the system achieves about 1 frame per second with three questions per frame. Performance with OpenAI's gpt-6-luna is slower at 0.2 FPS, likely due to connection overhead per question.
4Technical architecture
The Python implementation handles both local and cloud-based models through a unified interface. For local models, it uses llama.cpp's chat completions endpoint, while for OpenAI it adapts to their Responses format. The system processes webcam frames continuously while a background worker scores each frame. It includes special handling for different question types (boolean, multiple choice, ordinal scores) and normalizes the returned probabilities for consistent interpretation across question formats.
5Setup requirements
Running the system requires specific setup steps. For local execution with Gemma, users need to download the model and multimodal projector files (totaling ~7.2GB). The implementation uses llama.cpp compiled for CUDA architecture 86 (compatible with RTX 3090). The Python script depends on OpenCV for webcam access but doesn't use its vision capabilities - it's purely for frame capture. The system can alternatively connect to OpenAI's API, requiring an API key but no local model downloads.
6Performance considerations
The implementation makes several performance optimizations. It sets a small capture buffer size (1 frame) to minimize latency. Webcam frames are processed in a background thread while new frames continue to be captured. The system measures and displays throughput for evaluated frames, including image encoding time. For local models, it leverages llama.cpp's efficient execution on NVIDIA GPUs. The design avoids redundant image encoding by processing attachments once for all questions.
7Question format flexibility
The wrapper supports three question types with automatic formatting. Boolean (noul) questions return probability of true. Choice questions select among predefined options with individual probabilities. Score questions calculate weighted averages across ordinal levels. The system validates that each question has 2-20 criteria and represents them as lettered options (A-T). This structured approach maintains consistency while allowing diverse question types through the same interface.
8API compatibility layer
The implementation bridges differences between model providers' APIs. For OpenAI, it uses the Responses endpoint and constructs a specific request format including image URLs. For llama.cpp, it uses Chat Completions with a different image attachment syntax. The wrapper normalizes the probability outputs despite varying API response formats. It also handles API-specific requirements like OpenAI's need for including message.output_text.logprobs and llama.cpp's logprobs parameter.
9Practical applications
While specialized vision models would be more efficient, this approach offers unique advantages. The plain-text condition descriptions allow rapid prototyping without model retraining. Potential use cases include accessibility tools, smart environment monitoring, or rapid visual QA systems. The framework could be extended to other modalities beyond vision by adapting the attachments handling. The implementation demonstrates how LLMs can serve as flexible, if not maximally efficient, solutions for multimodal analysis tasks.
Original article: Hacker News — Building a Jev-like wrapper for LLMs with vision capabilities