Getting big results from a tiny model

The curator is the part of The Librarian that decides what gets remembered and structures it for efficient retrieval.

It runs three jobs:

• Harvesting picks durable facts out of a settled conversation transcript.
• Intake takes one proposed memory and decides whether to create, augment, supersede, split, archive or do nothing
• Grooming reviews slices of the corpus and tries to improve them.

I have been running my personal instance on DeepSeek V4 Flash, which is cheap and competent and has never given me a reason to complain. I wanted to move to a smaller open model to provide users (including myself) a smooth path to privacy through locally owned weights, and potentially a zero maintenance affordable hosted offering someday.

So the question was whether I could get a small, fast, open source model to do the job well enough.

I started with a quick shootout between 5 candidates:

• Qwen3.5-4B
• Qwen3.5-9B
• Gemma 4 12B Instruct
• Ministral 3 8B Instruct (the 2512 release)
• Granite 4.1 8B

Qwen3.5-9B was selected as the best combination of speed and performance.

Generating realistic and useful test cases

I started by having GPT 5.6 Sol generate test cases based on some clear guidelines by orchestrating locally hosted models that wrote the prose and reviewing their output. It did exactly what I asked, but the output was horribly formulaic: across one 100-case batch of fake vault slices 80% of the input memories were a single sentence about 14 words long. 95% of cases mentioned "project", 65% "decided", 52% "durable", 51% "constraint". Nothing in my actual vault is written like "a decided and currently active recoverable constraint". Technically valid, but it would have taught the model to correctly process content that looks nothing like what it will encounter in real use.

Next I tried having Sol turn real Claude transcripts and memories from my own vault into evaluation cases. The first batch passed every structural check but was completely useless because they were too easy to get right. One harvesting transcript more or less told the agent what to remember. An intake case offered only one plausible related memory and one grooming case presented an empty corpus.

The worst thing is that most of the selected candidates were just boring statements about the environment and dependencies. A fact has to add something the codebase cannot cheaply tell you to be worth remembering.

In the end what worked well was putting real anonymised transcripts in whole rather than as hand-picked excerpts, including all their natural corrections, digressions and dead ends. From these, synthetic siblings were rebuilt as counterfactual pairs (near-identical twins with different meanings). Same project, same names, same surrounding conversation, same length, but one deliberate difference that changes what the curator ought to do. Easy to get wrong during testing, and effective as training data because it teaches & tests the reasoning rather than the topic.

To guarantee quality I needed to actually review the cases, so I had Sol knock up a small local review site. It shows the evidence the curator saw and the outcome production actually applied, but hides the model, provider, prompt variant and expected answer until I've told it what I think of the outcome.

That review loop is where most of the real learning happened. It allowed me to mark a grooming no-op as wrong and not merely harmless, because "it did no damage" and "it did the necessary work" are not the same thing.

It also enabled me to quickly give specific per-case feedback which allowed Sol to make mistakes and learn from them. Nearly every improvement below started as a note typed into that page.

First the prompts

There are four prompts: a shared core, and job-specific sections for harvesting, intake and grooming. I built a loop for Sol to follow - change one job's prompt, run only that job's frozen cases against both candidate models, review blind, keep the change only if the aggregate improved, and note any new types of failure it introduces.

The scoring methodology was important too of course, and was different per-job. For example, harvesting used a single score that combines two things that pull against each other so neither can be gamed (known as F1):

• Precision: Of the memories the curator proposed, what fraction were actually worth keeping? This catches over-harvesting, the model shovelling in every trivial fact it can find.

• Coverage: Of the facts genuinely present in the transcript, what fraction did it find?

The original harvesting prompt had two specific faults. It asked for "Durable facts" which was too vague, so anything still true next week qualified, and "one fact per entry" actively encouraged fragmentation, splitting a decision away from the reason for it.

Neither model was struggling to read the transcripts, but both were selecting far too much trivially code-recoverable, transient and context dependant material.

The replacement asks for intent, rationale, meaningful history and future direction, and rejects facts cheap to recover from code, config, comments or tests. It also stops treating the speaker (human or agent) as a value signal since most of a technical conversation is usually written by the agent.

The same prompt work moved Qwen 35.2 points and Flash 35.7. I had been treating prompt engineering as a way of coaxing a weaker model up towards a stronger one, but the problems those prompts fixed had almost nothing to do with the model. Interestingly Flash retained its roughly 4% absolute advantage, although it became relatively less significant at their increased scores.

Intake and grooming moved too, but much less evenly.

Intake shifted least in that round, with Qwen going from three fully correct cases out of ten to four and Flash from five to seven because its prompt was carrying competing instructions and a brittle schema. A later round of intake-only prompt work eventually took the Qwen base to eight out of ten.

Grooming is where Qwen travelled furthest, but by the test metrics is still the least trustworthy. In practice though, I have found it makes subjectively much better decisions resulting in better recall. The biggest improvement has been greater consolidation of memories into single documents (by project and type of information for example). Recall is limited to the top 8 ranked memories by default (agents can override this, but would need to somehow know there is more needed). This caused overly fragmented memories to drop out of the returned results meaning agents would either fail to retrieve potentially important context or need to know to run further more focused requests.

Eventually the loop stopped realising meaningful gains because the remaining failures weren't because "the model has not been told what to do". It had been told, but was applying the rubric inconsistently.

Training

I did the LoRA adapter training on my home-built rig with two 16Gb Nvidia GPUs (5070Ti + 5060Ti), using Unsloth Studio over TRL (Hugging Face's Transformer Reinforcement Learning library).

Sol used the same case compiler to build training sets distinct from the evaluation cohort with no shared case, family, prompt or entity. Three LoRAs, one per job, all rank and alpha 8 (untouched model weights & default "volume"), one epoch (a single run), 8,192-token window (the cap for input prompt & output combined), completion-only loss (learn only from output, not the prompt) on Qwen3.5-9B.

Grooming came first because it is the most consequential job. Three adapters at 100, 250 and 500 cases produced training losses of 0.4865, 0.3806 and 0.3032 respectively. The blind results were nine corpora improved with none worse, then seven improved and two worse, then four and four. Counter intuitively the larger adapters produced worse Curators because they had learned to merge more aggressively, which is precisely the failure I was trying to remove. I think this was because they were learning the synthetic writer's habits rather than the judgement. The 100-case adapter was selected and took exact gold matches on the sealed cohort from 33 to 38, with prose drift falling from 43 cases to 35.

Harvesting worked too, and improved precision and coverage together instead of trading one for the other. Trained in just 7 minutes 38 seconds, it took harvesting F1 from 78.2 to 80.8, and both base and adapter returned exactly 65 candidates so the gain came from better selection.

Intake did not work. The mechanics were sound: validation loss fell, the adapter was demonstrably active, and a blind comparison mildly preferred it, two cases to one with seven ties. Base and adapter both scored eight out of ten. Nothing was broken, but it didn't deliver a measurable improvement.

The gains from fine-tuning were modest, but frankly I'm amazed such tiny datasets moved the needle at all. It was a fun learning experience, and I'll revisit it in a year or so when my personal Librarian corpus is much larger and more varied.

Did 9B beat Flash?

No, because Flash improved too with the prompt engineering. But that tiny Qwen model did finish far beyond where Flash was performing when I first started for all three jobs, so it is absolutely good enough to use now.

A tuned 9B parameter model running on a desktop gets close enough to a 284B hosted model to make privacy an easy choice over the small gain in accuracy. And it's a lot faster too.

I hope you enjoyed reading this, and that it inspires you to try some of these ideas in your own work. The autonomous prompt engineering loop and human reviewed synthetic test data are things I will definitely be taking into other projects.

If you'd like to see what the upgraded Curator can do for yourself, please give The Librarian a try.