thenug.net

giving claude code a memory palace

April 9, 2026

i was burning 45,000 tokens before i even said hello. every single Claude Code conversation loaded the same pile of memory files - session history, server details, past decisions - whether i needed them or not. most of that context was dead weight for any given conversation.

MemPalace fixes that by moving the bulk of your memory into a local vector database and only pulling what's relevant when it's actually needed.

the problem

Claude Code uses markdown files in ~/.claude/rules/ as persistent memory. every file in there gets injected into the system prompt at the start of every conversation. it works, but it doesn't scale.

my setup had grown to about 40KB of rules files:

that's roughly 10-12K tokens loaded every conversation. ask Claude to rename a file? here's 12K tokens of server architecture first. it's like reading an entire notebook before answering a yes or no question.

what mempalace is

MemPalace is a Python tool that creates a ChromaDB vector database from your files. it mines your docs, splits them into searchable chunks ("drawers"), and exposes them through an MCP server that Claude Code can query on demand.

instead of "load everything always," it's "search for what you need."

the architecture looks like this:

~/.claude/rules/          (always loaded, ~1.5KB now)
       |
       v
Claude Code context
       |
       v
MemPalace MCP server  <-->  ChromaDB at ~/.mempalace/palace/
       |                          |
       v                          v
  semantic search            5,452 drawers
  on demand only             mined from ~/docs/

the setup

installed via uv tool install mempalace to keep it isolated from system Python. on Arch, system Python was 3.14 but MemPalace needs 3.9-3.12, and uv handled that automatically by pulling cpython 3.13.

uv tool install mempalace

init creates the palace structure and lets you define entities (projects, people, etc.) that it should track:

mempalace init ~/docs

fair warning - the auto-detection for entities is rough. it detected Gitea and Gluetun as "people" and "User" as a project. skip the auto-detected ones and add your own manually.

mining took under 2 minutes for 430 files and produced 3,131 drawers:

mempalace mine ~/docs

wiring it into claude code

the MCP server registration is one command, but the Python path matters. since i used uv tool, the venv Python is at a specific path - not the system one:

claude mcp add mempalace -- \
  ~/.local/share/uv/tools/mempalace/bin/python \
  -m mempalace.mcp_server

the docs say python -m mempalace.mcp_server but that'll fail if your system Python doesn't have it installed. match the path to however you installed it.

MemPalace also ships two hooks that go in ~/.claude/settings.json:

the hook scripts aren't bundled in the pip package. grab them from the GitHub repo and drop them in ~/.mempalace/hooks/.

the slim-down

with MemPalace handling retrieval, the always-loaded rules files could be gutted. everything in them was already mined into searchable drawers.

before:

memory-sessions.md    27.5KB
memory-servers.md      5.9KB
memory-decisions.md    5.7KB
total:               ~40KB (~10-12K tokens)

after:

memory-sessions.md    ~0.2KB  (pointer to MemPalace)
memory-servers.md     ~0.8KB  (just IPs and API keys)
memory-decisions.md   ~0.2KB  (pointer to MemPalace)
total:                ~1.5KB  (~400 tokens)

kept a small quick-reference block (frequently used connection details) in the always-loaded file since those come up nearly every session. everything else - session history, detailed configs, past decisions - lives in MemPalace now.

gotchas

back up first. i took a full backup before touching anything:

tar czf ~/claude-memory-backup-full-20260409.tar.gz \
  ~/docs/ ~/.claude/

840MB but worth it. if MemPalace turns out to be the wrong call, restoring the old rules files is a 30-second operation.

the mempalace mcp subcommand doesn't exist in the current pip release despite being referenced in some docs. just wire up the MCP server manually.

chromadb pulls in onnxruntime for local embeddings. it's not a small dependency. if you're tight on disk or care about install size, be aware.

entity detection needs hand-holding. the NER is not great. manually define your projects and entities during mempalace init rather than trusting the auto-detect.

why not just use bigger context

token cost aside, there's a quality argument. when Claude's context is 60% background noise it didn't ask for, the signal-to-noise ratio drops. relevant information retrieved on demand beats irrelevant information loaded by default. it's the same reason you don't SELECT * when you need one column.

eli5

imagine you had a filing cabinet (your memory files) and every morning you photocopied the entire thing and stuffed it in your pocket before leaving the house (loading into context). most days you only needed one or two pages, but you carried all of it anyway. MemPalace is like leaving the filing cabinet at home but bringing a walkie-talkie (the MCP server) so you can ask someone to look up exactly what you need, exactly when you need it. your pockets (context window) stay light, and you still have access to everything.

← back to blog