Measured
A local qwen3.5:4b on 48 held-out synthetic cases agreed with human labels on 29 of 48 (60.4%), against an 85% bar. 2026-10-22.
Measured
How many extra tokens, and is the knowledge right.
The same tasks, run once with FULI and once without. Each number names its method, model, client and date.
In short
- FULI still adds input tokens: the latest run used 1.53× the input of no FULI, down from 2.50×.
- On tasks that need project knowledge, 18 of 18 runs succeeded with FULI and 3 of 18 without.
- On 32 held-out prompts the entry found 25; on 8 that should find nothing, it hit 2.
All on synthetic projects with small samples: read them as trends.
Tokens in a multi-turn session
A 6-turn coding session: how many times the input with FULI is the input without it.
- Input, on / off
- 1.53×Before the fix
2.50× - Cost, on / off
- 1.42×Before the fix
1.51× - Entry over 6 turns
- 24.3 KBBefore the fix
48.2 KB
The fix: within a session the entry sends only what changed, and fixed notes at the end of a prompt no longer steer recall. Merged; ships in 0.23.1.
Turn by turn
| Turn | Step | Entry bytes | Extra input share |
|---|---|---|---|
| 1 | Read | 8297 → 7969 | 9.2% → 19.3% |
| 2 | Change | 8281 → 2825 | 36.4% → 15.0% |
| 3 | Test | 8259 → 3481 | 44.8% → 34.1% |
| 4 | Change | 8297 → 2614 | 57.4% → 21.6% |
| 5 | Correct | 8337 → 4345 | 78.0% → 63.6% |
| 6 | Change | 6750 → 3096 | 77.4% → 17.8% |
Extra input share = the input FULI added over no FULI, as a share of the input with FULI. Before the fix on the left, after on the right.
How it was measured
- Client: Claude Code 2.1.301, headless. Turn 1 starts a session and each later turn resumes it.
- Model: claude-haiku-5-5. Date: 2026-11-06, one run before the fix and one after.
- Task: a 6-turn session on a small synthetic project: read the code, make three changes, run the tests, correct once.
- Scale: 2 sessions with FULI and 2 without. The FULI side uses a separate test instance with synthetic data only.
- Input = uncached input, cache reads and cache writes of every model request. Cost is the list price Claude Code reports.
Caveats
- The no-FULI side ran longer in one measurement than the other, so compare ratios and shares, not raw totals.
- Most of the extra input is cache reads, so cost rises less than input.
- Turn 5 is still high: the model loaded tools once more and made one more closing call.
- Neither side finished a whole session successfully, so there is no cost per successful task yet.
Is the delivered knowledge right
FULI searches once on every prompt. This checks whether it found the knowledge the task needed.
In the multi-turn sessions
| Check | Before | After |
|---|---|---|
| Right knowledge delivered on the three change turns | 0/6 | 4/6 |
| Turns that delivered unrelated items | 10/12 | 0/12 |
| Final checks passed at session end (with FULI) | 0/6 | 3/6 |
Held-out prompts
| Version | Found | Old one too | False hits |
|---|---|---|---|
| Before the entry recall change | 12/32 | 6/32 | 2/8 |
| After | 25/32 | 13/32 | 2/8 |
Both false hits gave titles only, no knowledge content. Choosing between an old and a new statement is left to the model.
How it was measured
- Held-out set: a different synthetic project, 40 prompts in Chinese and English. 32 each have one fact to find; 8 should find nothing. The prompts were written before reading the recall code.
- Only FULI's task entry runs, with no model, once per prompt. Dates: 2026-11-05 before the change, 2026-11-06 after.
- The multi-turn rows come from the same sessions as the token numbers above, counted by what the model actually saw.
Caveats
- The error-code fact was missed both times: a ranking gap, not fixed yet.
- A bare "continue" with no topic is missed by every search method.
With the knowledge delivered, did the task get done
Each of 6 tasks needs one project fact the repository lacks or states in an old form. Hidden checks judge the result.
| Setup | Done | Median tokens |
|---|---|---|
| No FULI | 3/18 | 43k |
| With FULI | 18/18 | 63k |
| Right fact written into the prompt (reference) | 18/18 | 39k |
With FULI matched being handed the answer, but used 47% more tokens than no FULI. The 3 successes without FULI were all the task whose answer the code reveals.
How it was measured
- Client: OpenCode. Model: Claude Opus 5.5. Date: 2026-11-04.
- Scale: 6 tasks, 3 runs per task per setup, 54 runs; failed runs count.
- Earlier that day, before the entry recall fix, with DeepSeek flash: 6% without FULI, 72% with FULI, 100% with the fact given.
- Remembering a new decision: 6 of 6 became current knowledge (awaiting your confirmation). Revising an old fact: 6 of 6 replaced the old version with a new one.
Is Tonborg's review accurate
Before knowledge is kept, Tonborg reviews it: keep, drop, or ask you. This checks whether its calls match human labels.
Strong-model re-run
A re-run on gpt-5.6-sol is in progress. Its result and method will be added here.
This is agreement on synthetic cases, not a result from real reviews.
Earlier measurements
In the early rounds FULI usually added tokens, and the knowledge often did not arrive. The numbers above were taken after those fixes.
Claude Code on a Mac, the same three-turn maintenance task
| FULI version | Date | Extra with FULI |
|---|---|---|
| 0.20.0-beta.12 | 2026-10-25 | +215% |
| 0.20.0-beta.15 | 2026-10-25 | +15% |
| 0.21.0-beta.1 | 2026-10-26 | +33% |
| 0.21.0-beta.2 | 2026-10-27 | +15% |
| 0.21.0-beta.3 | 2026-10-27 | +46% |
Claude Code 2.1.280, model claude-opus-5. Input plus output tokens as the client reports them. One run per version, so the spread is wide.
Two other rounds
- 2026-11-01 · OpenCode + Claude Opus 5.5 · 6 small tasks, 2 runs each, empty knowledge base: 17% more tokens with FULI, 11/12 done versus 12/12 without.
- 2026-10-25 · OpenCode + DeepSeek v4.1 flash · 11 merged changes redone, 66 runs: 19/33 done with FULI, 16/33 without; the knowledge barely arrived.
Not measured yet
- Everything ran on synthetic projects, not on real user data.
- Only 2 multi-turn sessions per side, too few to show spread.
- Behaviour after a long session is compacted is not measured.
- Multi-turn was measured only with Claude Code and Haiku 5.5 on Windows; not with Codex, Cursor or OpenCode.
- The OpenCode entry still sends everything on every turn; Codex uses the same incremental entry but was not measured.
- Tonborg's review accuracy with a strong model is still being measured.