Syllabus Lesson 51 of 156 · Build a blocker
Build a blocker

Precision and recall

Why it matters Every blocker makes two kinds of mistake: it lets trackers through, or it blocks something a page needs and breaks the site. Saying "my blocker works" means nothing until you can put a number on both. These two numbers are how researchers, and browser makers, compare blockers honestly.

A fishing net

Picture a fishing net meant to catch only mackerel. Two questions decide if it is any good:

  • Of everything in the net, how much is mackerel? (A net full of dolphins is a bad net, even if it caught all the mackerel.)
  • Of all the mackerel in the sea, how many did the net catch?

The first question is precision. The second is recall. You need both: a net with no holes catches every mackerel and every dolphin too.

The four boxes

Sort every request in the log into one of four boxes:

BlockedAllowed
Really a trackerTrue positive (tp): caught, goodFalse negative (fn): a privacy leak
Not a trackerFalse positive (fp): possible breakageTrue negative (tn): fine

Precision: tp / (tp + fp): of what you blocked, the share that really were trackers. Low precision means broken pages.

Recall: tp / (tp + fn): of all the tracker requests, the share you blocked. Low recall means leaks.

A worked example. A blocker stops 40 requests: 36 to trackers and 4 to the harmless CDN. Another 12 tracker requests got through. Precision is 36 / 40 = 0.9. Recall is 36 / 48 = 0.75. So: when it blocks, it is usually right, but a quarter of the tracking still happens.

Edge cases need a decision. If the blocker blocked nothing, precision would be 0 / 0. We say 1.0: nothing was blocked wrongly. If the log has no tracker requests at all, recall is also 1.0: nothing leaked. Writing such conventions down is part of measuring honestly.

The truth comes from somewhere

You can only score against an answer key: the set of sites that really are trackers. In the practice world we know it. In real research, people build it by hand, site by site, which is slow and imperfect. Always ask who decided what counts as a tracker.

A common misconception

"Higher recall is always better." Blocking every third-party request gives perfect recall, and it breaks logins, payments, maps and videos. People then switch the blocker off and end up with no protection. A slightly lower recall with high precision often protects more people in practice.

Try it Many blocker extensions show a count of blocked requests per page. Pick a site that misbehaves with the blocker on, then switch it off for that site and see what starts working. You just found a false positive.

In other languages

JavaScript:

function score(log, trackers) {
  const s = { tp: 0, fp: 0, fn: 0, tn: 0 };
  for (const r of log) {
    const t = trackers.has(siteOf(r.url)), b = r.outcome === "blocked";
    s[t && b ? "tp" : b ? "fp" : t ? "fn" : "tn"]++;
  }
  s.precision = s.tp + s.fp ? s.tp / (s.tp + s.fp) : 1;
  s.recall = s.tp + s.fn ? s.tp / (s.tp + s.fn) : 1;
  return s;
}

Go:

precision := 1.0
if tp+fp > 0 {
    precision = float64(tp) / float64(tp+fp)
}
recall := 1.0
if tp+fn > 0 {
    recall = float64(tp) / float64(tp+fn)
}

Recap

  • Precision: of what you blocked, how much was really tracking (low means breakage).
  • Recall: of all tracking, how much you blocked (low means leaks).
  • State your edge-case conventions, and know where the answer key came from.
  • Blocking everything maximises recall and ruins precision.
Your turn

Write score(log, trackers). For each record, it is a tracker request if site_of(record.url) in trackers and it was blocked if record.outcome == "blocked". Count tp, fp, fn and tn, and return a dict with those four plus precision (tp / (tp + fp), or 1.0 when nothing was blocked) and recall (tp / (tp + fn), or 1.0 when there were no tracker requests).

Spotted a problem in this lesson? Report it

Code · runs in your browser
Output

        

      

      
      

        Back