Is My Second Brain Actually Helping? Time to Benchmark. 🧠

In part one I built a local second brain and said it made working with my AI faster and cheaper. Here’s the uncomfortable thing: I didn’t actually know that. It felt faster. That’s a vibe, not a number.

So before I keep telling people this works, I wanted to prove it to myself. This post is about building something re-runnable that measures whether the local brain genuinely helps, instead of just trusting the warm feeling that it does.

What I’m actually trying to measure

I landed on four things, and they’re not all equally trustworthy. I want to be upfront about which is which.

1. Retrieval vs. stuffing. This is the one clean number. How many tokens does local retrieval save versus just pasting whole files into the prompt? It’s model-independent, so it doesn’t depend on which AI I happen to be using that day. But a savings number is worthless if retrieval is quietly failing, so I pair it with a recall@k check. That makes sure I’m never crediting “savings” to the system simply not finding the right note. And then a sufficiency check on top: did the slice it pulled actually contain a complete answer, or just the right source file with half the answer missing? Right file is not the same as right answer.

2. Cloud spend trend. Claude token usage before and after the local brain went live. The go-live cutline was 2026-06-18. I’ll say it plainly: this one is preliminary. A four-day window is noise. I need two to three weeks before this means anything, and I’d rather flag that now than wave around an early number that flatters my own project.

3. Local offload. How many tokens does qwen handle on-device, so the cloud model sees less? I think of this as an opportunity metric, not proof. Qwen is not Claude in quality, so offloading work to it is a tradeoff, not a free win.

4. Limitations. An actual section, not a footnote. Qwen isn’t Claude. Ratios aren’t absolutes. The sample is small. And the thing I’m measuring keeps changing, because the vault grows every session.

The harness

The benchmark lives in the vault under Benchmarks/, as Python scripts plus a generated dashboard note. A few choices I made on purpose:

For token counting I use Anthropic’s count_tokens API for the absolute numbers, never tiktoken, since I’m measuring Claude usage and want the real count. If there’s no API auth available it falls back to a char/4 estimate, and the ratios still hold even when the absolutes are approximate.

Claude usage gets parsed straight from the local ~/.claude/projects/.../*.jsonl logs, where the token usage sits on the assistant records. Since cwd is constant for me, attribution falls back to gitBranch.

The chunking in the harness strips frontmatter, splits on headers, keeps slices around 512 tokens (hard max 768, with 64 overlap), and never splits a fenced code block. And I cross-check the parser totals against npx ccusage so I’m not just trusting my own math.

The eval set is built to resist gaming

This is the part I’m most proud of, because it’s easy to write a benchmark that makes you look good. I tried to write one that fights back.

It’s ten grounded questions, each anchored to a “gold note” that has to show up in the top-k results for recall to count. A few of them are traps on purpose:

There’s a paired trap: two questions on the same topic (LTI launch, grade, auth) where the correct note is the opposite one (LTI 1.1 versus 1.3). Naive keyword matching fails one of them. Version-aware embeddings pass both. That’s exactly the kind of mistake I want to catch.

There’s a negative control: a question about something that isn’t in the vault at all (a Kubernetes cluster I don’t have). The correct behavior is low similarity and an honest “not in the vault,” not a confident hallucinated answer. I score that one separately as abstained true or false, and I keep it out of the recall denominator so it can’t pad the score.

And there are a couple of synthesis questions where recall only counts if all the gold notes show up, not just one. My target is recall@6 of at least 7 out of 10.

The honest gap

Here’s what the benchmark still doesn’t do, and it’s the most important thing. It measures retrieval-versus-stuffing and it tracks spend, but my actual claim is “the local brain saves me money.” That claim needs a counterfactual: the same task done with the brain and without the brain, three or four matched pairs, head to head. That’s the experiment that directly tests the thing I keep saying. I haven’t built it yet. It’s the next thing I’m building.

Why local RAG is genuinely hard to benchmark

If this seems like a lot of scaffolding for a personal project, here’s why it has to be. Local RAG resists clean measurement in specific ways:

There’s no labeled ground truth out of the box, so you build the gold-note anchors yourself. Retrieval and generation fail separately, which is why recall@k has to isolate just the retrieval half. The right source isn’t the right answer, which is the whole sufficiency problem. Small private datasets are noisy, so a single run can mislead you. Local models add their own wobble, so part of what you’re measuring is the model’s limits and not your system design. And the target keeps moving, because the vault grows every session.

So the honest framing of this whole post is: I’m not proving the second brain is great. I’m building the tool that will tell me, run after run, whether it actually is, including the runs where it isn’t. That’s the part I think most people skip.

🧠 Second Brain series