← Learnings

A follower count is not a multiplier

The obvious way to float popular accounts up a user search is to multiply relevance by follower count. Do that and every brand-new account scores zero, however well it matches.

PART 1 OF 2

Part 1 is the ranking problem: what to do with a popularity signal so it helps instead of taking over. Part 2 is the storage problem: why the field type you pick decides how much of your index the query has to read.

Multiplying has no ceiling

Someone searches for alex. The account @alex is an exact username match — joined last week, zero followers.

The text scorer gives it a 10. Multiply that by the follower count and the document comes out at 10 × 0 = 0. A zero sorts below every document that matched at all, and vanishes completely if anything downstream drops results with no score.

Zero times anything is zero. No amount of matching fixes it.

Now hand that account ten followers, so nothing in the arithmetic is zero any more. It still loses.

@thedailyplate has a million followers and a bio reading recipes with alex, every Sunday. The text scorer gives that passing mention a 0.5, a twentieth of the exact match. Multiply both by followers and you get 100 against 500,000. The weak match wins by a factor of five thousand.

Two different failures, one cause.

A multiplier has no ceiling. Zero is only its floor.

Follower counts on a social product span six orders of magnitude, while BM25 scores live in single digits. Multiply the two together and the popularity term decides the ranking outright, whatever the text scores were.

ONE QUERY: ALEX MULTIPLY — TEXT SCORE × FOLLOWERS @ALEX · 10 FOLLOWERS 100 @THEDAILYPLATE · 1,000,000 500,000 — off the chart, and on top of your results ADD — TEXT SCORE + BOOST, BOOST CAPPED AT 3 @ALEX · 10 FOLLOWERS 10.01 @THEDAILYPLATE · 1,000,000 3.49 TEXT MATCH FOLLOWER BOOST same two users, same two text scores — only the combining rule changed
FIG 1 · THE SAME TWO USERS, TWO RULES

What adding does instead

Turn the follower count into points capped at three, and add them on. Ten followers is worth 0.01 of those points and a million is worth 2.99, which I will derive in a moment. So 10.01 against 3.49. The exact match wins, and the popular account still outranks every other weak match on the page.

The whole difference is that the popularity term now has a maximum, and the maximum is a number you pick. A cap of three points means popularity can reorder documents whose text scores sit within three points of each other, and nothing beyond that. Lucene's own description of the field calls this a linear combination of a query-dependent score and a query-independent onesrc.

Linear is the load-bearing word.

One more thing the shape buys you. The text query goes in must and the boost in should, so a popular account with no textual match cannot be returned at all. Popularity is allowed to reorder the answers. It is never allowed to be one.

USER-SEARCH.JSONJSON
{
  "query": {
    "bool": {
      "must": [
        { "multi_match": {
            "query": "alex",
            "fields": ["username^8", "bio"]
        } }
      ],
      // optional: adds points, can never make a document match
      "should": [
        { "rank_feature": {
            "field": "followers",
            "boost": 3,
            "saturation": { "pivot": 2000 }
        } }
      ]
    }
  }
}
FIG 2 · THE WHOLE QUERY, AND THE CAP IS THE ONLY DIAL

Why the boost flattens out

Left alone, points-for-followers has the same problem in miniature: a million followers should not be worth a hundred thousand times what ten are worth. The saturation function handles that. It scores S / (S + pivot), always between 0 and 1, multiplied by your boostsrc.

pivot is the follower count that earns exactly half the boost. Below it you get less than half, above it more, and the curve flattens from there.

With a pivot of 2,000 and a cap of 3: ten followers collects 0.01 points, two thousand collects 1.5, a million collects 2.99. The input is a power law and the output is a bounded ramp.

At first glance that looks like a defect. An account with 200,000 followers and one with 2,000,000 end up within a hundredth of a point of each other — and that is the intent. Past a certain popularity, more popularity stops telling you anything about which of two matches the person actually meant.

Supply no pivot and the engine picks one for you: roughly the geometric mean of the values in the indexsrc. That is a reasonable place to stand before you have data to tune with.

BOOST = 3 × S / (S + 2000) 3.0 CAP 0 1.5 +0.01 PIVOT — HALF THE CAP +2.99 1 100 2K 10K 100K 1M followers, each step ten times the last — the boost cannot leave the band
FIG 3 · A POWER LAW IN, A BOUNDED RAMP OUT

Where the zero-follower user ends up

A rank_feature field takes strictly positive values, and rejects negatives and arrayssrc. An account with no followers has nothing to put in it, so you leave the field out of the document entirely.

The should clause then doesn't match that document at all. It collects no boost, and it keeps every point of its text score. The absence of a boost is not a penalty. The new user who matched perfectly is still sitting on 10.0, which is precisely what multiplication took away.

ASIDE

Some features correlate the wrong way — account age, URL length, distance. Set positive_score_impact: false and saturation flips to pivot / (S + pivot), so the score falls as the value rises. Then make sure every document carries a value, because a missing negative feature reads as no penalty, which is backwardssrc.

What a boost is not

It is not a sort. If someone asks for most-followed first, a boost is the wrong tool — that is a sort clause, and sorting by a field switches relevance scoring off altogether. A boost exists to break ties between good matches, not to order a leaderboard.

The cap turns out to do a second job I did not expect: it is also what makes the query fast. That is part 2.

The one-paragraph version

Popularity belongs in a search score as a bounded number of points added to the text score, never multiplied into it, because a multiplier has no ceiling: a million followers buys a million-fold advantage and zero followers buys annihilation. Cap the boost at a few points, run the raw count through a saturation curve so that a celebrity and a semi-celebrity land in the same neighbourhood, and keep the text query in must so popularity can only ever reorder answers it did not choose. The cap is the whole idea. Everything after it is how you spend it.