Detect trackers without a list

Spotting a shoplifter without a photo

A shop guard with a list of known shoplifters is useful. But a good guard also watches behaviour: someone who visits every shop on the street and writes down who goes in. You do not need their name to see what they are doing.

Trackers have tell-tale behaviour too. In this lesson you look for three clues in a browser's request log:

1. Everywhere at once. The same third-party site shows up on two or more different sites. That is what lets it join your visits together.
2. A unique ID in the address. A query value like cid=df0641532d7f that looks random: that is a name tag for you.
3. A pixel. An "image" of 100 bytes or less, too small to show anything. Its only job is to make your browser send a request.

Our rule: a site is a tracker if clue 1 is true and at least one of clues 2 or 3 is true. Requiring clue 1 keeps the harmless cdn.example safe: it is on every shop, but it sends no ID and its logo is a real 2 KB picture.

How random is random? Entropy, briefly

To tell an ID from a normal word, we measure how mixed-up its characters are. aaaa is not mixed at all. abcd uses four different characters equally.

char_entropy("aaaa")   # 0.0  every character is the same
char_entropy("abab")   # 1.0  two characters, half each
char_entropy("abcd")   # 2.0  four characters, a quarter each

math.log2(p) is "how many times do I halve 1 to get p", given as a negative number, which is why there is a minus sign. collections.Counter(text) counts each character for you. You will meet this same formula again, applied to whole crowds of browsers, in the next module.

Entropy alone is not enough: trail-shoes is quite mixed too. So looks_like_id also asks for at least 8 characters, only letters, digits, - or _, and at least one letter and one digit. Words fail the digit test; dates and URLs fail the letter or character test.

Reading the log

Each record in browser.log has what you need:

• site_of(record.url) and record.top_site: if they differ, it is third party.
• record.request.params: the query as a dict, already decoded.
• record.response: None if blocked; otherwise .header("Content-Type") and .size in bytes.

Honest limits

A tracker on only one site in your log passes unnoticed, even if it is on thousands of sites you have not visited yet. A tracker that hides IDs in a cookie and sends a normal-sized image also slips through. And a first-party analytics script that sends IDs to its own site is not third party at all. That is why real blockers use lists and heuristics together.

function charEntropy(text) {
  if (!text) return 0;
  const counts = {};
  for (const ch of text) counts[ch] = (counts[ch] || 0) + 1;
  let bits = 0;
  for (const n of Object.values(counts)) {
    const p = n / text.length;
    bits -= p * Math.log2(p);
  }
  return bits;
}

const looksLikeId = v => v.length >= 8 && /^[A-Za-z0-9_-]+$/.test(v)
  && /[0-9]/.test(v) && /[A-Za-z]/.test(v) && charEntropy(v) >= 2;

static double CharEntropy(string text) =>
    text.Length == 0 ? 0 :
    -text.GroupBy(c => c)
         .Select(g => (double)g.Count() / text.Length)
         .Sum(p => p * Math.Log2(p));