Stanford researchers have released Memorilla, an open-source memory system for LLMs. The Memorilla memory system reads a whole document collection and hands a frozen LLM just 16 memory tokens. On FactKG, a frozen Qwen3-8B scored 84.33% with Memorilla, against 80.43% with every document in context. It did that with 108x fewer document tokens. Code, training recipes and checkpoints are public under Apache 2.0. You can check the deployment options and specs here through our GPULeaderboard.com chrome extension.
TL;DR
- Size: 194M-parameter trainable memory module. It always outputs 16 memory tokens, whatever the collection size. The encoder (Qwen3-Embedding-4B) and decoder (Qwen3-8B) stay frozen.
- Runs on: Linux with CUDA GPUs and Python 3.11. The quickstart needs 1 GPU with 80 GB (recipes ran on H100 80GB). No quantized variants disclosed.
- Performance: Highest reported average across 8 benchmarks (49.68), top score on 5 of 8.
- Best: PersonaMem-v2 accuracy of 44.60%, versus 27.92% for full context.
- Worst: PubMedQA accuracy of 59.00%, versus 69.00% for LLMLingua-2.
What is Memorilla?
Memorilla is a memory module that turns a document collection into 16 vectors for a frozen LLM. The work comes from Jure Leskovec’s SNAP group, with co-authors from Toronto, Penn and MIT.
A frozen LLM can only use what sits in its context window. Full-context prompting grows slower as history grows. Retrieval pastes the chunks most similar to the question, and can miss weaker clues. Memorilla takes a third route: it learns what each question needs from the whole store.
How does Memorilla work?
The pipeline has 3 steps:
- Encode once. A frozen Qwen3-Embedding-4B embeds each document a single time. New documents never force re-encoding of old ones.
- Read per question. 16 learned slots attend to the question and the full store. The module uses 2 cross-attention layers with 8 heads.
- Inject. The slots are projected into Qwen3-8B’s input embedding space and prepended to the question.
Only the memory module trains: 194,123,776 parameters for the 8B decoder. Training runs in 3 stages. Stage 1 uses about 1.98M Wikipedia paraphrase-reconstruction examples. Stage 2 adds about 1.34M examples from 10 QA and summarization datasets. Stage 3 fine-tunes on all 8 benchmarks. The full recipe takes about 19 hours on 4 H100 GPUs.
How does Memorilla perform on benchmarks?
All baselines use the same frozen Qwen3-8B, per the paper’s main table. Scores are self-reported by the research team.
- FactKG: 84.33% accuracy, versus 80.43% full context and 80.38% MemoRAG.
- PersonaMem-v2: 44.60%, versus 30.90% for xRAG, the next best method.
- TriviaQA: 57.00 exact match, just ahead of MemoRAG at 56.60.
- LaMP-4 / LaMP-7: 18.15 and 48.37 ROUGE-L, both the best scores.
- Average: 49.68, versus 41.92 for xRAG, 40.75 for LLMLingua-2 and 38.02 for Mem0.
Memorilla loses on 3 benchmarks. On PubMedQA it scores 59.00%, while LLMLingua-2 reaches 69.00%. Those abstracts total only 324 tokens, so compression has little to gain. MemoRAG leads on PersonalizationV4 (67.88% vs 66.65%) and NarrativeQA ROUGE-L (20.51 vs 19.31). MemoRAG used about 690x and 50x more document tokens there.
Compression peaks on NarrativeQA: 16 tokens replace 67,385 document tokens, a 4,212x cut. As agent memory, 8 memory tokens lift a Qwen3-4B TextWorld agent from 18.6% to 24.2% average success.