flowchart LR Q[Claim] --> K[Keyword search<br/>BM25] Q --> D[Meaning search<br/>embeddings] K --> F[Rank fusion] D --> F F --> S[Top 100<br/>candidates] S --> R[Referee<br/>17-feature reranker] R --> T[Top 10<br/>to the model]
Ask a language model a question, and in a retrieval-augmented generation (RAG) system it doesn’t answer from memory alone. It first goes to a library, pulls out a handful of pages and reads them, then answers.
Generating the answer is the easy part. Picking the pages is harder. Give the model the wrong pages and it will summarise them confidently anyway.
This post follows one project that improves the fetching in layers, with each small idea built on the last. The final layer is a learned reranker. Training it means solving a family of quadratic programs, and that’s where Jaddle comes in.
NoteThe short version
- Keyword search scores 0.681. Adding embeddings and fusing the two rankings gets 0.731.
- A 17-feature reranker, trained as a ranking SVM, lifts that to 0.770: about 4 points, with a 95% interval of +2.1 to +6.0.
- Jaddle solves the reranker’s whole training path, 11 quadratic programs, in one batched GPU call in about 3 seconds.
Keeping score
The test bed is SciFact, a set of scientific claims paired with the research abstracts that support or refute them. Every score in this post is nDCG@10 on its held-out test claims. The metric looks only at the top ten results. It gives more credit for a relevant abstract near the top than near the bottom, and a perfect ranking scores 1. A gain of 0.01 is one “point”.
The pipeline at a glance
Layer 1: Match the words
The classic starting point is BM25. Search for “does coffee help memory?” and it scores each page by how often coffee, help and memory appear in it, with two refinements:
- Rare words count for more. Matching caffeine is far more telling than matching the.
- Long pages don’t win by bulk. A page can’t win just by containing a lot of words.
BM25 is fast, simple and hard to beat. It scores 0.681.
Its blind spot is vocabulary. If the claim says car and the abstract says automobile, BM25 can’t tell they’re related.
Layer 2: Learn which words matter
SciFact comes with training labels: for each training claim, the abstracts that are actually relevant. We can use them to tune BM25.
Give every word its own multiplier, starting at 1. Then adjust the multipliers so that, for each training claim, relevant abstracts outscore irrelevant ones by a margin. Words that point towards the right answer get boosted, and words that are mostly noise get turned down.
That lifts the score to 0.708. It’s better, but the helper still only sees words.
Layer 3: Learn what the words mean
A text embedding turns a passage into a list of numbers (384 of them here, from the compact bge-small model). Passages with similar meanings get similar lists, so car and automobile end up close together even though they share no letters.
To search, we embed the claim the same way and look for abstracts whose vectors point in the same direction. This is dense retrieval, and on its own it scores 0.720, already ahead of the tuned keywords.
Keyword search and dense retrieval make different mistakes, so it pays to combine them. Reciprocal rank fusion does this using only each page’s position in the two rankings. An abstract that ranks well in both rises to the top. Because it uses positions, it never has to reconcile two scores on different scales. The fused ranking scores 0.731. Its top 100 also holds 96.5% of the relevant abstracts, up from 92% for BM25’s top 100.
Layer 4: A referee for the shortlist
We don’t have to trust any single ranking. Take the fused top 100 as a shortlist, and bring in a referee to reorder it.
The referee looks at 17 features of each (claim, abstract) pair:
- Keyword signals: BM25 on the full text, on the title alone and on exact word forms, TF-IDF, and the learned term-weight score from Layer 2
- Meaning signals: embedding similarity, plus an older technique called LSA
- Coverage: how much of the claim the abstract mentions, including runs of consecutive words
- Context: the lengths of the claim and the abstract, and the fused rank itself
Every feature means the same thing for every claim. “High embedding similarity” is good news whatever the topic, so what the referee learns on training claims carries over to claims it has never seen.
The referee itself is simple: multiply each feature by a weight and add them up. The question is what the weights should be.
Training the referee
For each training claim, we pair every relevant abstract with five irrelevant ones from the same shortlist, which gives about 3,600 pairs. We want two things:
- In every pair, the relevant abstract should score at least 1 point higher than the irrelevant one.
- The weights should stay small, so the referee doesn’t memorise the training claims.
These two goals pull against each other, so we allow each pair a slack \(\xi_t\) for falling short and charge a price for using it. With weights \(v\) and feature vectors \(x_t^+\) and \(x_t^-\) for the two abstracts in pair \(t\), the problem is
\[ \min_{v,\ \xi}\ \ \tfrac12 \|v\|^2 + \frac{C}{T}\sum_{t=1}^{T} \xi_t \quad \text{subject to} \quad (x_t^+ - x_t^-)^\top v \ \ge\ 1 - \xi_t, \qquad \xi_t \ge 0 . \]
This is a ranking SVM, and it is a quadratic program: a quadratic objective under linear constraints. The constant \(C\) sets the balance. A large \(C\) insists on getting the pairs right, while a small \(C\) prefers simpler weights.
Where Jaddle comes in
One quadratic program is easy enough. But we don’t know the right \(C\) in advance, so we solve the problem for 11 values of \(C\) and keep whichever does best on a set of validation claims.
\(C\) appears only in the objective’s linear term. Every problem on the path therefore has the same structure, and Jaddle solves all 11 side by side in a single batched call on the GPU. Each problem has about 3,600 variables, and the whole path takes about 3 seconds. As a check, the chosen problem is solved again with Clarabel, an interior-point solver, and the optimal values agree to about 1e-5.
Since the whole path is this cheap, there’s no need to guess \(C\). We can try every value and keep the best. The same trick fits the term weights in Layer 2.
Results
| Stage | nDCG@10 | Change |
|---|---|---|
| BM25 keyword search | 0.681 | |
| Learned term weights | 0.708 | +2.7 |
| Dense retrieval (embeddings) | 0.720 | +3.9 |
| Rank fusion of BM25 and dense | 0.731 | +5.0 |
| Referee, 14 keyword features only | 0.724 | +4.3 |
| Referee, all 17 features | 0.770 | +8.9 |
The full referee beats rank fusion by 4.0 points (95% interval +2.1 to +6.0, from a paired bootstrap over test claims). The validation set agrees: 0.817 against 0.791.
The embeddings do two jobs:
- They widen the net. Fusion’s shortlist contains relevant abstracts that BM25’s top 100 misses.
- They give the referee a signal keywords can’t. Without the three embedding features, the referee loses to plain fusion (0.724 against 0.731).
The learned weights show what the referee relies on. Embedding similarity carries the most weight. The learned term score and coverage come next, contributing the precision of exact keyword matches. LSA gets no weight at all, because the embeddings already provide everything it offers.
WarningA fair comparison
The referee is trained on SciFact’s own labelled claims. Many published retrieval results are zero-shot: the models never see the target dataset’s labels. Read 0.770 as evidence that this recipe works when you have labelled examples. It is not a general claim about how this pipeline compares with every other search system.
The takeaway
Good retrieval is built in layers:
- Match the words. BM25 is a strong, cheap baseline.
- Learn which words matter from labelled examples.
- Add meaning with embeddings, so car and automobile finally get along.
- Bring in a referee to reorder the shortlist, and let a fast batched solver choose its weights.
None of these layers is clever on its own. Stacked together, they pick the right pages more often, which gives the model a better chance of answering correctly.
The full example is examples/cp/rag_reranker.py in the Jaddle repository.