I Built a Local Second Brain. Now My AI Never Forgets. 🧠

Every time I opened a new Claude Code session, it started completely blank. No memory of the decisions we made yesterday, no context on the project, just a fresh slate. So I’d re-explain everything. Again. And every time I did that, I was burning time and burning tokens to get back to where we already were the day before.

That got old fast. So I built a fix. A local, private second brain that gives my AI persistent memory of my projects, with 100% of the data staying on my machine.

The problem, plainly

The context window is not your memory. It fills up, older stuff gets pushed out, and the next session doesn’t inherit any of it. For a one-off question that’s fine. For an ongoing project, it means you’re the memory. You carry the history in your head and re-feed it every session. That doesn’t scale, and honestly it’s the most boring part of the work.

I wanted the project itself to remember. Not me.

The fix: a three-layer stack

The whole thing runs on three pieces, and you own all of them.

🧠 Obsidian is the storage layer. It’s just a local markdown vault, plain text files on disk. Nothing in the cloud. My vault is called “Hugolocal.”

⚡ Ollama runs open-source models directly on my machine. I use qwen2.5:14b for chat and reasoning, and nomic-embed-text for embeddings. This is the local tier that does search and retrieval without anything leaving the laptop.

🤖 Claude Code is the heavy-reasoning collaborator. It runs as a plugin in my VS Code, and it reads and writes notes in the vault as we work.

To be honest about it: the stack is not 100% local. Claude Code is cloud. The way I think about it is a two-tier setup. A local tier handles retrieval, search, and routine summarizing, all private. A cloud tier handles the heavy reasoning. The goal is to push as much as I can down to the local tier so the cloud tier sees less.

The memory loop

This is the part that actually makes it work. There’s a CLAUDE.md file sitting in the vault that tells Claude Code how to behave:

1. At the start of a session, read the relevant notes.
2. At the end, write a session log into a Sessions/ folder.
3. Keep one canonical status.md updated, the single source of truth for the project.

So the session logs become the history, and status.md is current reality. The important bit is that these truths live on disk. They survive the context window filling up, because they were never trapped in the context window to begin with. New session starts, Claude re-grounds itself from the vault, and we pick up exactly where we left off.

I structured it PARA-style, which is a pattern I borrowed from Tiago Forte’s Second Brain work: Projects/, Areas/, Resources/, Archive/, plus my Sessions/ folder. And I applied this to an existing project, not a clean empty folder. It worked on the real thing.

Where the savings come from

Normally you re-explain everything or paste giant chunks of code into every session. That’s token-heavy and it gets worse as the project grows.

With the second brain, Claude reads targeted, distilled notes instead. A 500-word status.md can replace thousands of tokens of re-explanation. Only the relevant context loads. So as the project gets bigger, the cost stays flat or even shrinks, because I’m not re-feeding the whole history every time. That’s the compounding effect, and it’s the part I didn’t fully appreciate until I saw it happen.

The honest part

I want to be careful not to oversell this. A few things I’m watching:

Within a single long session, this setup still hits the same pruning problem. The vault doesn’t auto-reinject status.md halfway through, so if a session runs long enough, earlier truths can still scroll out. The complete answer probably combines externalized truth on disk with a proactive reread when context fills up. I’m not all the way there yet.

And “it feels faster” isn’t proof. It feels faster. That’s a vibe, not a number. Measuring whether this genuinely helps is its own project, and that’s where part two goes.

Try it

If you’ve got a GPU with 8GB+ of VRAM, or a Mac with 16GB+ of unified memory, you can have a version of this running in well under an hour. Pull Ollama, set up an Obsidian vault, drop in a CLAUDE.md, and write your first few notes. Everything stays on your machine, and you own every piece of it.

Your data stays local. Your context compounds.

🧠 Second Brain series