# JioHotstar Benchmarks Knowledge Bases for AI Coding Agents JioHotstar evaluates retrieval strategies to optimize AI agent performance in complex codebases. - **Source:** [InfoQ](https://brifing.si/source/68ac18af-a4f3-4f7c-9c2c-44d4af8f89dd) - **Published:** 2026-11-11 - **Topic:** [technology](https://brifing.si/c/technology) - **Original article:** - **On Brifing:** https://brifing.si/s/c8f66448-3b05-4062-bc9c-e5701c91a515 ## Evaluating AI Agent Retrieval JioHotstar conducted a comprehensive evaluation of six distinct approaches for assisting AI coding agents in navigating large codebases. Senior Staff Engineer Abhishek Bansal compared filesystem tools, Sourcebot, Graphify, CocoIndex Code, Repowise, and an internal RAG pipeline. The study utilized Claude Sonnet as the primary model to measure task accuracy and efficiency. ## Performance Across Different Benchmarks The team utilized subsets of SWE-bench and CodeScaleBench alongside a custom question-answering harness for their internal codebase. Results indicated that filesystem baselines remained highly competitive for concrete engineering tasks. However, Sourcebot outperformed these baselines on more complex SWE-bench Verified tasks while simultaneously reducing overall operational costs for the development team. ## Structural Versus Semantic Retrieval Structural code search proved most effective for clearly defined, specific engineering tasks. Conversely, semantic retrieval provided significant advantages when agents faced ambiguous requirements that required mapping concepts to implementation details. The findings suggest that developers should select retrieval strategies based on specific task characteristics rather than relying on a single universal solution. ## Tool Design and Agent Preference The evaluation highlighted a clear preference among agents for familiar developer operations, such as searching text, reading files, and locating symbols. Agents frequently bypassed higher-level abstractions that required additional verification steps. This behavior suggests that tool design should prioritize intuitive, standard developer workflows to ensure higher reliability and better task completion rates. ## Limitations of Retrieval Layers Adding complex retrieval layers did not consistently improve task completion or reduce token consumption across all scenarios. On simpler SWE-bench Lite tasks, these layers often introduced unnecessary overhead without providing measurable performance gains. This indicates that over-engineering the retrieval process can sometimes hinder agent efficiency rather than enhancing the final output quality. ## Beyond Codebase Repository Indexing Siddharth Vaithinathan argued that repository indexing alone is insufficient for high-level architectural tasks. He suggested that agents must understand the company context, including historical architectural decisions, constraints, and past incidents. Capturing the rationale behind design choices is essential for agents to propose meaningful changes in complex, legacy-heavy software environments. ## Measuring Tool-Use Traces JioHotstar emphasizes the importance of monitoring tool-use traces alongside standard task accuracy metrics. By analyzing how agents interact with tools, teams can determine if retrieval methods actually reduce unnecessary exploration. This granular approach to performance measurement helps distinguish between effective agent reasoning and inefficient, repetitive tool calls that inflate operational costs. ## Strategic Recommendations for Architects The report concludes that no single retrieval approach dominates all coding tasks. Architects should maintain filesystem tools as a reliable baseline while selectively integrating structural search for code-focused problems and semantic indexing for cross-cutting requirements. Matching retrieval capabilities to the specific nature of the programming task remains the most effective strategy. ## Sources covering this story - https://www.infoq.com/news/2026/11/jiohotstar-codingagent-retrieval/?utm_campaign=infoq_content&utm_source=infoq&utm_medium=feed&utm_term=global — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._