Syllabus Lesson 146 of 156 · Measure the web like a researcher
Measure the web like a researcher

Crawl the simulated web

Why it matters Counting is how you turn a vague worry ("trackers are everywhere") into a fact you can check and repeat. In this lab you run a real-style crawl over 1,000 simulated sites and build the table every tracking study starts with: which third parties appear, and on how many sites.

A crawl is like a stock check in a shop: walk every aisle, write down every item, then add up. Here each aisle is a site, and each item is a request the page made.

The simulated web

SimWeb(seed) from watched.leg7b_web makes 1,000 fictional sites, ranked by popularity. web.sites lists their names and web.visit(site) returns what an instrumented crawler recorded:

from watched.leg7b_web import SimWeb, TRACKER_LIST
from watched.urls import site_of

web = SimWeb(seed=1)
load = web.visit(web.sites[0])
print(load["site"], load["rank"], load["category"])
for r in load["requests"][:4]:
    print(r["kind"], r["host"], r["sets_cookie"])
print(site_of("t.adco.example"))       # adco.example

Third party: a site other than the one you are visiting that the page contacts. If you visit calmriver.example and it loads t.adco.example, the third party is adco.example.

Registrable domain (site): the part of a host name that one owner controls, such as adco.example for t.adco.example. site_of(host) works it out for you; you wrote it yourself in Leg 6.

Prevalence: how widespread something is: here, the number (or share) of crawled sites on which a third party appears.

Counting with a dict, sorting with a key

A set keeps each value once, which is perfect for "which third parties did this page contact". To count across pages, use a dict and add one for each page a third party appears on. To sort the table, sort its items with a key: a function that says what to sort by.

table = {"adco.example": 3, "cdn.example": 5, "fontshelf.example": 3}
ranked = sorted(table.items(), key=lambda kv: (-kv[1], kv[0]))
print(ranked)   # [('cdn.example', 5), ('adco.example', 3), ('fontshelf.example', 3)]
# -kv[1] puts big counts first; kv[0] breaks ties alphabetically.

Note that a third party counts once per site, however many requests it makes there. A tracker that fires ten pixels on one site is still on one site.

TRACKER_LIST is a blocklist of known tracker sites, the way public filter lists label them. The share of sites that load at least one listed tracker is the headline figure of many studies.

A common misconception: "Every third party is a tracker." Fonts, content delivery networks and payment widgets are third parties too. Your table will show some of the most common third parties are harmless, which is why studies label carefully.

In other languages

Counting prevalence in JavaScript (Node) and Go:

// JavaScript
const table = new Map();
for (const load of loads) {
  const thirds = new Set(load.requests.map(r => siteOf(r.host)).filter(s => s !== load.site));
  for (const s of thirds) table.set(s, (table.get(s) || 0) + 1);
}
const top = [...table].sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])).slice(0, 10);
// Go
table := map[string]int{}
for _, load := range loads {
    seen := map[string]bool{}
    for _, r := range load.Requests {
        s := siteOf(r.Host)
        if s != load.Site && !seen[s] { seen[s] = true; table[s]++ }
    }
}

Recap

  • A crawl visits many sites and records every request; counting turns that into facts.
  • Use site_of to group hosts into sites, and count each third party once per site.
  • Sort with a key to rank by prevalence, breaking ties by name.
  • Not every third party is a tracker; label with a list and say which list you used.
Your turn

Write five functions:

  1. third_parties(load): the set of sites (site_of(request["host"])) the page contacted, without load["site"].
  2. prevalence(loads): a dict from third-party site to the number of loads it appears in.
  3. top(table, n): the n largest (site, count) pairs, biggest first, ties alphabetical.
  4. cookie_setters(loads): the set of third-party sites with at least one request whose sets_cookie is true.
  5. tracker_share(loads, tracker_list): the fraction of loads that contact at least one site in tracker_list.
Then fill loads with web.visit(site) for every site in web.sites, in order. Tests also crawl other simulated webs.

Spotted a problem in this lesson? Report it

Code · runs in your browser
Output

        

      

      
      

        Back