JioHotstar Benchmarks Knowledge Bases for AI Coding Agents
JioHotstar evaluates retrieval strategies to optimize AI agent performance in complex codebases.
1Evaluating AI Agent Retrieval
JioHotstar conducted a comprehensive evaluation of six distinct approaches for assisting AI coding agents in navigating large codebases. Senior Staff Engineer Abhishek Bansal compared filesystem tools, Sourcebot, Graphify, CocoIndex Code, Repowise, and an internal RAG pipeline. The study utilized Claude Sonnet as the primary model to measure task accuracy and efficiency.
2Performance Across Different Benchmarks
The team utilized subsets of SWE-bench and CodeScaleBench alongside a custom question-answering harness for their internal codebase. Results indicated that filesystem baselines remained highly competitive for concrete engineering tasks. However, Sourcebot outperformed these baselines on more complex SWE-bench Verified tasks while simultaneously reducing overall operational costs for the development team.
3Structural Versus Semantic Retrieval
Structural code search proved most effective for clearly defined, specific engineering tasks. Conversely, semantic retrieval provided significant advantages when agents faced ambiguous requirements that required mapping concepts to implementation details. The findings suggest that developers should select retrieval strategies based on specific task characteristics rather than relying on a single universal solution.
4Tool Design and Agent Preference
The evaluation highlighted a clear preference among agents for familiar developer operations, such as searching text, reading files, and locating symbols. Agents frequently bypassed higher-level abstractions that required additional verification steps. This behavior suggests that tool design should prioritize intuitive, standard developer workflows to ensure higher reliability and better task completion rates.
5Limitations of Retrieval Layers
Adding complex retrieval layers did not consistently improve task completion or reduce token consumption across all scenarios. On simpler SWE-bench Lite tasks, these layers often introduced unnecessary overhead without providing measurable performance gains. This indicates that over-engineering the retrieval process can sometimes hinder agent efficiency rather than enhancing the final output quality.
6Beyond Codebase Repository Indexing
Siddharth Vaithinathan argued that repository indexing alone is insufficient for high-level architectural tasks. He suggested that agents must understand the company context, including historical architectural decisions, constraints, and past incidents. Capturing the rationale behind design choices is essential for agents to propose meaningful changes in complex, legacy-heavy software environments.
7Measuring Tool-Use Traces
JioHotstar emphasizes the importance of monitoring tool-use traces alongside standard task accuracy metrics. By analyzing how agents interact with tools, teams can determine if retrieval methods actually reduce unnecessary exploration. This granular approach to performance measurement helps distinguish between effective agent reasoning and inefficient, repetitive tool calls that inflate operational costs.
8Strategic Recommendations for Architects
The report concludes that no single retrieval approach dominates all coding tasks. Architects should maintain filesystem tools as a reliable baseline while selectively integrating structural search for code-focused problems and semantic indexing for cross-cutting requirements. Matching retrieval capabilities to the specific nature of the programming task remains the most effective strategy.
Original article: InfoQ — JioHotstar Benchmarks Knowledge Bases for AI Coding Agents