Hushkey: Silent Dictation via Local Lip-Reading on macOS
A privacy-focused, local-first silent speech recognition tool using webcam-based lip and throat tracking.
1Silent Dictation via Webcams
Hushkey enables silent dictation by tracking lip, jaw, and throat movements through a standard webcam. Unlike audio-based systems, it processes visual data locally to convert mouthing into text. This proof-of-concept tool runs entirely on macOS, ensuring that no video or audio data ever leaves the user's local machine environment.
2Architecture and Local Pipeline
The system utilizes a native Mac application that communicates with a local model server. It captures visual input, processes it through a conformer-based lip reader, and outputs text directly at the cursor. The pipeline includes adapter training, allowing users to personalize the model to their specific mouth movements and articulation patterns.
3Kinematic Data and HushNet
HushNet integrates a kinematic branch that monitors lip aperture, jaw drop, and throat motion via vertical optical flow. It employs a zero-initialized gate, ensuring the model only utilizes these extra streams if they improve accuracy. This approach enhances the base lip-reading performance by incorporating physical cues beyond simple RGB pixel analysis.
4Personalization and Model Training
Users can personalize the system by recording approximately 40 silent prompts, which trains a small adapter file on their Mac. The training process includes sequence-level policy gradient reinforcement learning to minimize word error rates. This ensures the model adapts to individual hypo-articulation and specific tempo habits during silent speech input.
5Privacy and Security Measures
Privacy is maintained by discarding all video frames immediately after processing. Only enrollment clips, stored as mouth crops and keypoint features, are saved in the local library folder. The software does not require cloud connectivity, and users can optionally integrate local LLMs via Ollama for text cleanup and formatting.
6Technical Requirements and Setup
Hushkey requires macOS 14 or later, with Apple Silicon hardware highly recommended for performance. The installation involves a model server download of approximately 1.5 GB. Developers can build from source using provided scripts, while standard users can utilize the DMG installer, though manual permission overrides for unsigned binaries are currently required.
7Current Limitations and Accuracy
The base model reports a 19.1% word error rate on LRS3 datasets, but real-world silent mouthing is significantly more challenging. Visually identical sounds like /p/, /b/, and /m/ remain difficult to distinguish without context. Users are advised to exaggerate movements slightly and dictate phrases rather than single words to improve decoding.
8Future Roadmap and Development
Planned improvements include notarization for smoother macOS integration and Core ML export to leverage the Neural Engine, removing Python dependencies. The roadmap also explores adding AirPods head-motion input and TrueDepth sensor support. These updates aim to transition the project from a proof-of-concept toward a more robust, production-ready silent dictation utility.
Original article: Hacker News: Show HN — Hushkey: Silent Dictation via Local Lip-Reading on macOS