Building a Camera-Contextual Voice AI on Android
How Live Lens combines real-time vision and voice for interactive AI assistance
1Multimodal Android demo
Live Lens is an experimental Android application that combines real-time camera input with voice AI interaction. Users point their phone at a physical workspace while conversing with an AI agent that understands visual context. The demo uses Jetpack Compose for UI, Agora for real-time voice infrastructure, and Gemini models for both conversational AI and visual understanding. This creates a seamless experience where the phone serves as the agent's eyes during live sessions.
2Technical architecture
The system employs a clear separation between Android client and Python backend. The mobile app handles camera preview, microphone input, and UI interactions through Agora RTC. The backend manages sensitive operations including Agora token generation, Gemini API calls, and agent lifecycle control. Visual context is captured periodically via camera snapshots, processed by Gemini Vision, and injected into the ongoing voice conversation. This hybrid approach maintains security while enabling visual awareness.
3Key Android implementation
The Android application implements several critical voice AI components: permission handling for camera/microphone, Agora RTC channel management, real-time event processing via RTM, and Compose UI state management. The demo showcases practical patterns like keeping credentials off-device, modeling the voice session as observable state, and integrating camera controls within the communication stack. It uses Agora's preview instead of CameraX since camera operations are part of the RTC session.
4Security considerations
The architecture deliberately avoids embedding sensitive credentials in the Android app. All Agora tokens are generated server-side, and Gemini API keys remain exclusively on the backend. The client only receives short-lived RTC/RTM tokens and communicates with the backend via a configurable URL. This pattern is crucial for production Voice AI apps where model selection, agent configuration, and user authorization typically reside server-side.
5Session state management
The demo treats the Voice AI session as comprehensive UI state in Compose, encompassing listening status, visual context updates, session duration, and control states. This centralized approach avoids scattering session logic across callbacks and enables coherent rendering of the interactive interface. The state includes camera controls (switch, torch, zoom), mute functionality, manual context refresh, and session termination capabilities.
6Practical vision integration
Rather than streaming continuous video to the AI model, Live Lens adopts a pragmatic frame capture approach. The Android app takes periodic snapshots which the backend summarizes using Gemini Vision, then injects as context into the voice session. This balances computational efficiency with contextual awareness, making the agent feel visually informed without requiring complex video processing pipelines or excessive bandwidth usage.
7Development workflow
The project supports rapid iteration on physical devices through ADB port forwarding, allowing direct backend access via localhost. This facilitates real-world testing of permission flows, audio routing, and camera behavior early in development. The setup requires Android Studio, Python 3.10+, an Agora project with Conversational AI enabled, and a Gemini API key - all detailed in the repository's setup instructions.
8Production potential
While experimental, Live Lens demonstrates a viable foundation for commercial applications in field support, education, and troubleshooting scenarios. The core loop of visual context capture and voice interaction could extend to device diagnostics, repair assistance, or laboratory guidance. Future enhancements might include smarter frame selection, visual memory retention, annotation overlays, and deployment optimizations for scaled usage.
Original article: ProAndroidDev - Medium — Building a Camera-Contextual Voice AI on Android