Knowledge Graph Verification via Crowdsourced Feedback
A novel framework refining graph relationships through real-time search interactions.
1Challenges in Knowledge Graphs
Knowledge Graphs (KGs) built by Large Language Models (LLMs) often create incorrect relationships based on statistical text co-occurrence. For instance, 'Pho' might be wrongly classified under 'Italian Noodle Soup.' Such errors lead to irrelevant search results and reduced user trust. Manual verification doesn't scale for millions of dynamically changing entities in domains like food delivery or e-commerce.
2Traditional Verification Limitations
Conventional methods rely on human annotators or domain experts, which are slow and expensive. They can't keep up with daily changes in restaurant menus or product catalogs. When inaccurate edges enter the graph, ranking algorithms propagate these errors, degrading user experience. For example, incorrect parent-child links result in poor query expansion and lower Click-Through Rates (CTR).
3User Feedback-Driven Validation
The new framework treats unverified graph edges as hypotheses. These edges are injected into search suggestions and tested against real user interactions. The system balances showing verified results (e.g., 'Sushi' under 'Japanese Cuisine') with exploring unverified candidates. This ensures users mostly see accurate results while the system collects validation data for new relationships.
4Hierarchical Relationship Refinement
Unlike traditional systems that only validate entities (nodes), this framework validates structural links (edges). It confirms whether entity A is truly a parent, child, or sibling of entity B. For example, it tests if 'Pho' should be under 'Vietnamese Cuisine' based on user engagement patterns, ensuring the graph's structural integrity.
5Risk-Managed Injection Strategy
The system injects candidate edges into low-risk UI positions, like the 3rd or 4th slot in 'related searches' carousels. To prevent frustration, it limits the number of candidates per session. Users primarily interact with verified content, while the system gathers enough data to validate new hypotheses without degrading search quality.
6Behavior Tracking Module
The system tracks user micro-interactions with high granularity. It records not just final clicks but also the search context (anchor terms) in which interactions occur. For instance, a click on 'Pho' is logged alongside the original query that led to it. This contextual data helps determine whether a candidate relationship reflects true semantic intent.
7KG Core Architecture
The central repository stores entities (e.g., dishes, products) and their connections. Each edge is classified as either verified (validated by high traffic or human review) or candidate (LLM-generated hypotheses). Candidate edges remain invisible to main search algorithms until validated. For example, 'Sushi → Japanese Cuisine' is a verified edge used in standard searches.
8Continuous Validation Loop
The framework operates as a closed-loop ecosystem: 1) Hypothesis generation (new candidate edges), 2) Controlled injection into UI, 3) Signal aggregation from user interactions, 4) Graph updates based on validation scores. This iterative cycle separates AI hallucinations from semantic truths using real-world engagement data.
9Adaptive Exploration Policy
The system employs a greedy exploration strategy. It exploits known good results (e.g., verified edges) to maintain user trust while exploring unverified candidates to gather validation data. For example, it might test multiple parent candidates for 'Pho' until user interactions confirm the correct categorization under 'Vietnamese Cuisine.'
10Industry Applications
This framework is particularly valuable in high-velocity domains like food delivery or e-commerce, where catalogs change daily. For instance, when a new dish is added, its LLM-generated relationships can be quickly validated via user interactions. Early adopters report 30% reductions in irrelevant search results and improved CTR by leveraging this continuous validation mechanism.
11Performance Metrics
Success is measured via CTR improvements and user satisfaction surveys. Initial tests show 15-20% of unverified edges are incorrect. Correcting these errors reduces irrelevant search results by up to 30%. Users increasingly trust platforms that deliver accurate results, leading to higher retention rates. For example, one food delivery app saw a 25% increase in repeat orders after implementation.
12Future Potential
The approach isn't limited to food or product categorization. It can be applied to any dynamic knowledge graph, such as music genres on streaming platforms or topic classification in news aggregators. Continuously improving graphs through user interactions could redefine semantic search, making it more adaptive and accurate across industries.
Original article: Grab Tech — Kullanıcı Geri Bildirimleriyle Bilgi Grafiği Doğrulama