Maha complete evidence
62.8%
95% CI 56.7–68.6%
[ Original benchmark // MCRB-1 ]
How much evidence survives a fixed context budget?
MCRB-1 evaluates six extractive selectors on 250 independently annotated questions across 136 full scientific papers. The primary result is exact and auditable: did at least one complete human evidence set survive?
Mean token reduction
74.4%
1,508.7 output tokens
Front truncation
25.6%
2.45× lower complete-set retention than Maha
Known-evidence ceiling
99.6%
36.8 points of headroom remain
Measured comparison
Maha leads every deployable baseline tested.
All methods received the same fixed selection allowance and returned passage-level citations. The oracle uses gold labels and is shown only as an upper bound.
| Method | Complete evidence | Any evidence | Mean recall | Reduction | p95 local |
|---|---|---|---|---|---|
| Maha BM25 | 62.8% | 72.4% | 67.4% | 74.4% | 5.91 ms |
| Maha keyword | 58% | 68.4% | 63% | 74.4% | 5.86 ms |
| Front truncation | 25.6% | 30.4% | 27.9% | 73.1% | 2.97 ms |
| Tail / recency | 20.4% | 25.6% | 22.8% | 73.3% | 2.52 ms |
| Seeded random | 22% | 33.6% | 27.3% | 73.3% | 2.7 ms |
| Oracle ceiling | 99.6% | 99.6% | 99.6% | 73.3% | 6.86 ms |
Economics
The measured savings exceed the machine fee.
- Mean input
- 6,447.1 tokens
- Tokens avoided
- 4,938.4
- Gross cost avoided
- $0.014815
- $0.001 fee multiple
- 14.81×
Uses a declared reference input rate of $3/million tokens. Output generation cost is excluded equally. This is a workload result, not a universal savings promise.
Position robustness
Ranking matters after the introduction.
front evidence (95 cases)66.3% complete
middle evidence (93 cases)59.1% complete
back evidence (62 cases)62.9% complete
Front truncation retained 0% of complete evidence sets in both the middle and back buckets. Maha BM25 retained 59.1% and 62.9%, respectively.
Interpretation boundary
Retention is not answer quality.
MCRB-1 tests whether independently highlighted evidence remains available to a downstream model. It does not test whether that model reasons correctly, cites correctly, or tells the truth. Generative summarizers and LangChain summarization are excluded from v1 because exact-span scoring penalizes paraphrase, while an LLM judge would make the primary result model-dependent.
Apply the evidence
Turn the benchmark into an implementation boundary.
Reproduce and audit
Every case and measurement is public.
npm run benchmark:context-retention