bayesian-bm25: Bayesian BM25: Turning Search Scores Into Probabilities You Can Trust
How Cognica’s library converts raw BM25 into calibrated relevance signals, then uses log-odds fusion and Bayesian priors to make hybrid search less ad hoc.
Standard BM25 produces unbounded scores that lack consistent meaning across queries, making threshold-based filtering and multi-signal fusion unreliable. Bayesian BM25 addresses this by applying a sigmoid likelihood model with a composite prior (term frequency + document length normalization) and computing Bayesian posteriors that output well-calibrated probabilities in [0, 1].
- Bayesian BM25 is really a calibration project, not a scoring tweak, because it turns retrieval output into probabilities that can be thresholded and reasoned about.
- Its core move is to combine a likelihood model with priors and base-rate correction, so the score becomes a posterior instead of a raw ranking artifact.
- Once BM25 and vector similarity live on the same probabilistic scale, fusion stops looking like a hand-tuned compromise and starts looking like inference.
- The library’s value is as much practical as theoretical, because it aims at hybrid search pipelines that need stable, interpretable signals early in the stack.
BM25 Is Useful, but Its Scores Lie
BM25 is excellent at ranking. It is bad at telling you what a score means. The same number can imply very different things across queries, corpora, and retrieval settings, which makes thresholding and signal fusion feel improvised.
That is the opening move in Bayesian BM25. It does not argue that BM25 is broken. It argues that BM25 should not be treated like a probability when it is really only a rank signal.
Hybrid search practitioners struggle, not because of embeddings, but because BM25 fails to fit into clean expectations of a probability.
The Move: From Raw Score to Calibrated Probability
The repo’s central idea is simple to state and subtle to execute: start with a raw BM25 score, model it as a likelihood, fold in a composite prior, then apply base-rate correction to get a posterior probability. That is a very different claim from normalization. Normalization rescales. Bayesian inference updates belief.
That shift matters because a posterior can do things a ranking score cannot. You can threshold it. You can compare it across queries more safely. You can fuse it with another calibrated signal without inventing a new weighting scheme each time.
We introduce a corpus-level base rate prior that decomposes the posterior into three additive log-odds terms, reducing expected calibration error by 68-77% without requiring relevance labels.
What the Library Actually Computes
The heart of the implementation lives in probability.py. The main abstraction, BayesianProbabilityTransform, wires together the score model, a composite prior, and a base-rate prior so the final output is numerically stable and interpretable.
The details matter. The code clamps probabilities before taking logits, and it uses split-form exponentials to avoid overflow. That is not cosmetic. Search systems live or die on edge cases, and calibration is only useful if it survives real distributions.
# Conceptual flow, simplified
score = bm25_score(query, document)
likelihood = sigmoid(score)
prior = tf_prior(term_frequency) * norm_prior(document_length)
base_rate = corpus_prior(score_distribution)
posterior = update_bayes(likelihood, prior, base_rate)
return clamp_probability(posterior)The implementation also reflects a strong bias toward production safety. If a transform is mathematically elegant but brittle at the tails, it is not ready for search traffic.
Why Fusion Gets Interesting Once Everything Is a Probability
Once BM25 is calibrated, fusion stops being a weighted guessing game. In fusion.py, the project uses log-odds conjunction and related gating logic to combine signals in a way that preserves uncertainty instead of flattening it.
That is the real conceptual upgrade here. A heuristic like 0.7 * BM25 + 0.3 * vector is easy to write and hard to defend. A probabilistic conjunction at least tells you what kind of uncertainty you are combining.
| Method | Can threshold it? | Calibrated? | Interpretable across queries? | Fuses lexical and vector cleanly? | Complexity trade-off |
|---|---|---|---|---|---|
| Standard BM25 raw scoring | Not reliably | No | Weak | No | Low, but ad hoc |
| Manual normalization and weighted sums | Sometimes | Usually not | Weak | Sometimes | Low, but hand-tuned |
| Reciprocal Rank Fusion | No | No | Moderate for ranking | Good for rank lists, not probabilities | Low and robust |
| Bayesian BM25 | Yes | Yes, by design | Strong | Yes | Moderate, with more math upfront |
| Cross-encoder reranking | Yes, after rerank | Often yes, if trained well | Strong on reranked set | Not a fusion method | Highest compute cost |
The Hybrid Search Bridge
The library does not stop at lexical search. vector_probability.py extends the same probabilistic framing to dense retrieval, using likelihood-ratio ideas and index-aware priors so vector similarity can be compared on the same conceptual scale as BM25.
That is why the project feels bigger than a BM25 wrapper. It is trying to give hybrid search a common language. Once that exists, lexical and dense systems stop competing as incompatible scoring worlds.
Why This Is Better Than Manual Weight Tuning
The practical appeal is not subtle. If your retrieval signals are probabilities, you can reason about them the way you reason about any other model output. You can set thresholds, inspect confidence, and combine stages without pretending raw scores are interchangeable.
That also makes Bayesian BM25 a useful middle layer in the stack. It is earlier and cheaper than reranking, but more principled than rank-only fusion. It sits in the awkward but valuable space between heuristics and heavy models.
Origin, Validation, and Where It Fits
The project comes from Cognica and Jaepil Jeong, and the code reads like a reference implementation of a paper trail rather than a one-off library. The repository structure backs that up: typed Python, tests, benchmarks, docs, and examples all point in the same direction.
There is also real-world validation. The project shows up as a baseline model in MTEB and in txtai for score normalization. That is the difference between a clever theory repo and a tool people actually wire into systems.
