Open Benchmark Reveals Memory Layer Performance
Full chat history compared against memory-based approach using same model
1Memory Optimization Benchmark
LeapMemory's open benchmark measures the performance of memory layers in AI conversations. Identical questions are tested under two conditions: pasting full chat history versus LeapMemory's memory-based approach. This method aims to significantly reduce token usage while maintaining model accuracy, offering insights into efficient AI memory management.
2Testing Methodology Details
Each question is asked twice - once with complete conversation history, once with only LeapMemory's recall result. Both conditions use the same model with zero temperature setting. Answers are automatically evaluated against predetermined ground truth. Token counts come directly from provider billing, ensuring accurate measurement of computational efficiency.
3Results and Efficiency Gains
Tests across three datasets showed token reduction between 78.2% and 91.7% using LeapMemory's approach. A real 59-turn conversation demonstrated 82.6% efficiency improvement. The memory-based method becomes increasingly efficient compared to full history paste as conversations grow longer, highlighting its scalability advantage.
4Technical Architecture
LeapMemory performs LLM extraction on each conversation chunk to build a knowledge graph alongside raw data storage. Recall combines entity scanning, graph walking, and hybrid search techniques to construct effective context. The system uses Qwen3-Embedding-0.6B for embeddings, maintaining consistent production configuration across all queries without probe-specific tuning.
5Current Scope and Limitations
This benchmark currently tests only direct recall questions: identifiers, names, numbers, and dates. Complex capabilities like cross-referential reasoning, contradiction resolution, or temporal reasoning aren't yet measured. Standardized long-term memory evaluations (LongMemEval-class) are planned for future releases to expand the testing scope.
6Run Your Own Tests
Developers have shared the complete infrastructure as open source, enabling users to run benchmarks with their own datasets. Clone the GitHub repository, add your API key, and start testing. The system creates a fresh tenant for each run, deleting previous results to ensure consistent measurements and reliable comparison between memory approaches.
Original article: Hacker News: Show HN — Bellek Katmanlarının Performans Karşılaştırması Açıklandı