fuseResults: BM25-only candidates bypass weighting and clamp to 1.0, so keyword matches always outrank semantic ones #978

Summary

In hybridRetrieval, fuseResults() scores candidates differently depending on whether they had a
vector hit, and the BM25-only branch bypasses the configured weighting entirely:

const fusedScore = vectorResult
  ? clamp01(Math.max(weightedFusion, bm25Score >= 0.75 ? bm25Score * 0.92 : 0), 0.1)
  : clamp01(bm25Result.score, 0.1);   // <-- raw BM25 score, no weighting

BM25 scores are unbounded (Lucene-style), while cosine similarity is bounded to roughly 0–1.
clamp01 therefore maps essentially every BM25-only candidate to 1.0, while a
semantically-correct candidate is limited to vectorScore * vectorWeight — at most vectorWeight.

With the default vectorWeight: 0.7, a keyword-only match scores a perfect 1.0 and a strong semantic
match caps at 0.7. Keyword-only candidates systematically outrank semantic ones, regardless of how
vectorWeight / bm25Weight are configured, because the weighting is not applied on that branch.

Measured on a real store (4,316 memories, jina-embeddings 1024-dim)

Query: "what has Mike decided about failed message sends, and why?"

After fusion: BM25-only candidate → clamp01(25.86) = 1.0; best semantic candidate →
0.4279 * 0.7 = 0.2995.

Observed effect

The correct record ranks #2 of 4,316 by raw cosine similarity and is returned at #2 by
store.vectorSearch — but never appears in hybrid results. Instead, all ten returned results were
long auto-generated "Reflection" documents, most containing only empty section headers
((none captured)). Those 32 records are 1% of the store yet occupied 100% of the top
results for many different queries, because length gives them broad BM25 term coverage while their
semantic similarity is mediocre.

Downstream consequences we hit:

• A cross-encoder rerank cannot help — it only reorders what fusion passes it.
• The tier-1 suppression governance fires constantly (auto-recall skipped after governance filters,
  ~822 events), because the candidates genuinely are poor.

Workaround

Setting retrieval.mode: "vector" bypasses fusion and restores correct ordering — verified: the same
query went from 10/10 irrelevant to the correct record at #1. The cost is losing keyword matching and
the rerank stage, since vectorOnlyRetrieval does not call rerankResults.

Suggested fix

Normalise the BM25 score into the same range as cosine similarity before fusing (e.g. min-max over
the returned candidate set, or a saturating transform such as s / (s + k)), and apply
bm25Weight on the BM25-only branch as well:

: clamp01(normalisedBm25 * this.config.bm25Weight, 0.1);

Notes

• memory-lancedb-pro 1.1.0-beta.11, OpenClaw 2026.7.2-beta.5.
• Happy to supply the diagnostic script used to produce the numbers above.