Building Frontier AI Models: A Product Management Perspective
Tulsee Doshi discusses the strategic role of product managers in developing Google's Gemini.
1Beyond Pure Engineering
Tulsee Doshi, Head of Product for Gemini, argues that building frontier models transcends traditional engineering. She compares the process to a complex potions class, requiring a blend of scientific rigor and creative intuition. This approach ensures that multimodal models like Veo and Lyria function effectively within diverse, real-world user environments.
2The 60-20-20 Framework
To manage development cycles, Doshi utilizes a specific 60-20-20 framework. This strategy balances features that are currently functional, those nearing completion, and experimental capabilities. By segmenting development this way, teams can maintain a consistent release cadence while simultaneously pushing the boundaries of what AI models can achieve for end users.
3PM Role in AI Labs
Product managers in AI labs are increasingly vital for bridging the gap between raw research and consumer utility. Doshi emphasizes that PMs must now possess specialized skills in model evaluation and 'vibes' assessment. They are responsible for ensuring that technical outputs align with intended product goals and high-quality user experiences.
4Model Evaluation Mechanics
Building an effective evaluation, or 'eval,' is a core competency for modern AI product managers. These systems measure model performance across various metrics, including skin tone representation and overall response quality. Establishing these benchmarks early allows teams to iterate rapidly while maintaining safety standards and mitigating potential biases in outputs.
5Full-Stack AI Strategy
Google’s strategy involves a full-stack approach to frontier AI, integrating safety, monitoring, and cyberdefenses directly into the development lifecycle. This holistic method ensures that models like Nano Banana and Genie are not just powerful, but also secure and reliable. The focus remains on building a cohesive ecosystem rather than isolated tools.
6UX for AI Models
Doshi highlights that the model itself possesses a unique user experience that requires careful design. Product managers must evaluate the entire interaction flow rather than just the raw model output. This shift requires teams to consider how AI agents and swarms integrate into existing workflows to provide genuine, measurable value.
7Adapting to Rapid Change
As model capabilities evolve, product teams must remain highly adaptable to avoid obsolescence. Doshi suggests building products that can leverage future model improvements without requiring complete architectural overhauls. This involves creating flexible AI harnesses that allow for seamless transitions as underlying technology becomes more sophisticated, efficient, and capable over time.
8Future of AI Product
The role of the product manager is shifting toward orchestrating complex AI systems rather than just managing static features. Doshi emphasizes the importance of avoiding 'AI slop' by focusing on meaningful, high-utility work. As AI assistants become more personal, the ability to judge model quality will define successful product leadership.
Original article: Lenny's Podcast — Building Frontier AI Models: A Product Management Perspective