From Cloud LLM to On-Device Whisper: Speech to Structured Actions on Android
Processing speech and extracting actions with on-device Whisper instead of cloud solutions.
1Transition from Cloud to On-Device
The article explains the shift from cloud-based LLMs to on-device Whisper for processing voice commands. Privacy, offline capability, and cost advantages motivated this transition. While Firebase with Gemini has limitations, the benefits of local solutions are emphasized.
2Speech Recognition with Whisper.cpp
Whisper.cpp, a lightweight version of OpenAI's Whisper model, offers a C/C++ based solution. The 42MB quantized model runs efficiently on mobile devices. Integrated with Kotlin via JNI bridge, it represents true 'on-device AI' implementation.
3Deterministic Parser Design
In the cloud version, LLM handled both transcription and structured data extraction. The new system separates these steps: Whisper only transcribes while a Kotlin-based deterministic parser processes the text. This ensures predictability and testability.
4Architecture Decisions and Interface Consistency
The system is designed around SpeechToTransactionService interface, facilitating switching between cloud and on-device solutions. Hilt dependency injection isolates UI layer from implementation details. This approach provides architectural flexibility.
5Native Code Integration
JNI bridge integrates Whisper.cpp with Kotlin. A dedicated coroutine dispatcher handles thread safety. CMake and Gradle configurations support only arm64-v8a architecture to optimize APK size.
6Performance and Constraints
The small model was selected considering quality-performance balance. Thread-unsafe native code requires special management. The rule-based parser's lack of flexibility stands as the main trade-off.
7Security and Privacy Advantages
Keeping voice data on-device is crucial for sensitive scenarios like financial transactions. Fully offline processing maximizes user privacy. Long-term costs are also reduced.
8Real-World Application Scenario
In the sample app, a voice command like 'Send Maria 10 euros for the taxi' is converted into a structured transaction object. Action type, amount, receiver and currency fields are automatically parsed, significantly improving user experience.
9Learning Experience and Future
The author describes this as a learning exercise. With limited NDK/C++ knowledge, AI assistance was utilized. While not a production blueprint, it presents a valuable case study demonstrating on-device AI potential. Future integration of more sophisticated models is envisioned.
Original article: ProAndroidDev - Medium — Android'de Konuşmayı Yapılandırılmış Eylemlere Dönüştürme