VoiceMem

VoiceMem is an open-source memory system for real-time voice agents that uses a streaming dual-brain architecture to store factual and emotional data.

VoiceMem acts as a memory system designed for real-time voice agents by utilizing a streaming dual-brain architecture to manage information. The left brain handles factual memory through schema and entities, while the right brain manages personality, emotional attribution, and relationship nodes. It operates by continuously segmenting audio, transcribing speech, and extracting structured information while the user speaks. The system supports integration with models like Qwen2.5-Omni, Qwen3-Omni, and Step-Audio2-Mini to provide persistent context. By prioritizing performance, it uses speculative prefetching to achieve response times of 134 ms. This decoupled architecture allows for the replacement of individual components, including the underlying memory engine.

VoiceMem at a glance

Best for: Software engineer
Runs on: open source, self-hosted
Pricing: Free
Released: August 2026
Category: AI audio tool
Website: github.com/xzf-thu/VoiceMem

Key features

Streaming architecture for real-time memory extraction
Dual-brain storage separating factual and emotional data
Low-latency retrieval with 134 ms response times
Support for multimodal inputs including speech and sound events
Integration with Qwen and Step-Audio model families
Speculative prefetching for query performance
Configurable barge-in control during audio playback

Use it for

Building long-term memory for real-time voice agents
Managing user personality and emotional state in AI conversations
Storing multi-speaker dialogue context from live audio
Developing voice interfaces that recall past interactions

Alternatives to VoiceMem

SimpleMem — Provides semantic lossless compression to reduce token consumption and maintain long-term context.
memU — It gives every connected agent a shared and inspectable memory layer.
Mem0 — It provides drop-in persistent memory for AI agents without changing pipelines.
MemMachine — Provides long-term persistent memory for LLM-based agents.
Supermemory — Integrates long-term memory into AI coding agents with a single prompt.
ReMe — It enables local-first, file-native long-term memory sharing across multiple AI agents.
Memmy — Shares one persistent memory foundation across your different AI tools.
MemPalace — Achieves high recall with local-first storage and zero API calls.
Memvid — Single-file architecture replaces complex RAG pipelines with portable storage.
Agentmemoryrepo — Enables agents to navigate memories like a wiki and share knowledge across sessions.
AgentMemory — It runs locally with zero external databases and achieves high retrieval recall.
AssemblyAI — Industry-leading speech models deliver high accuracy and low latency for voice applications.

All VoiceMem alternatives