# Open Benchmark Reveals Memory Layer Performance Full chat history compared against memory-based approach using same model - **Source:** [Hacker News: Show HN](https://news.ycombinator.com) - **Published:** 2026-07-22 - **Topic:** [Data & AI](https://brifing.si/c/data-ai) - **Original article:** - **On Brifing:** https://brifing.si/s/cf153dc3-1b4c-433f-9dd9-a7d2be11dcad ## Memory Optimization Benchmark LeapMemory's open benchmark measures the performance of memory layers in AI conversations. Identical questions are tested under two conditions: pasting full chat history versus LeapMemory's memory-based approach. This method aims to significantly reduce token usage while maintaining model accuracy, offering insights into efficient AI memory management. ## Testing Methodology Details Each question is asked twice - once with complete conversation history, once with only LeapMemory's recall result. Both conditions use the same model with zero temperature setting. Answers are automatically evaluated against predetermined ground truth. Token counts come directly from provider billing, ensuring accurate measurement of computational efficiency. ## Results and Efficiency Gains Tests across three datasets showed token reduction between 78.2% and 91.7% using LeapMemory's approach. A real 59-turn conversation demonstrated 82.6% efficiency improvement. The memory-based method becomes increasingly efficient compared to full history paste as conversations grow longer, highlighting its scalability advantage. ## Technical Architecture LeapMemory performs LLM extraction on each conversation chunk to build a knowledge graph alongside raw data storage. Recall combines entity scanning, graph walking, and hybrid search techniques to construct effective context. The system uses Qwen3-Embedding-0.6B for embeddings, maintaining consistent production configuration across all queries without probe-specific tuning. ## Current Scope and Limitations This benchmark currently tests only direct recall questions: identifiers, names, numbers, and dates. Complex capabilities like cross-referential reasoning, contradiction resolution, or temporal reasoning aren't yet measured. Standardized long-term memory evaluations (LongMemEval-class) are planned for future releases to expand the testing scope. ## Run Your Own Tests Developers have shared the complete infrastructure as open source, enabling users to run benchmarks with their own datasets. Clone the GitHub repository, add your API key, and start testing. The system creates a fresh tenant for each run, deleting previous results to ensure consistent measurements and reliable comparison between memory approaches. ## Sources covering this story - https://github.com/leapmemory/memory-bench — --- _Summary written by Brifing from the original reporting. Copyright in the original article belongs to its publisher; read the full piece at the link above._