Audit a site's third parties

Think of a guest book at the cafe door. At the end of the day you read it and ask: who came in, how often, and did anyone leave something behind? A request log is the website's guest book.

One entry from the practice world looks like this (trimmed):

{"method": "GET",
 "url": "https://adco.example/p.gif?u=https%3A%2F%2Fshopb.example%2F",
 "kind": "beacon",
 "top_site": "shopb.example",       # the site in the address bar
 "status": 200,
 "set_cookies": ["adco_id=9f3c; Max-Age=31536000; Secure; SameSite=None"]}

Real HAR files do not have top_site; you would read it from the page URL. Our log adds it to keep the code short.

Three questions an audit answers

1. Is this request to a third party? Compare the site of the URL with top_site. Use the site, not the full host: img.shopb.example belongs to shopb.example, so it is first party.
2. How much does each outside host get? Count its requests and note whether it set cookies.
3. Who sees people across sites? A host that shows up under two or more different top_site values can link visits. Those are the ones to look at first.

Two helpers from the practice world do the URL work for you: host_of(url) gives the host ("img.shopb.example"), and site_of(url) gives the registrable site ("shopb.example").

from watched import host_of, site_of
url = "https://img.shopb.example/logo.png"
print(host_of(url))   # img.shopb.example
print(site_of(url))   # shopb.example

To build a count per host, the dict method setdefault(key, default) is handy: it returns the value for key, first storing default if the key is new.

counts = {}
for word in ["a", "b", "a"]:
    row = counts.setdefault(word, {"n": 0})
    row["n"] += 1
print(counts)   # {'a': {'n': 2}, 'b': {'n': 1}}

A common misconception: "a host that sets no cookies is harmless". A host that appears on every site still sees the IP address and browser each time, and can pass its own ID in the URL instead of a cookie. Count requests and sites, not only cookies.

// JavaScript (Node)
const har = JSON.parse(fs.readFileSync("page.har", "utf8"));
const report = {};
for (const e of har.log.entries) {
  const host = new URL(e.request.url).hostname;
  report[host] = (report[host] || 0) + 1;
}
console.table(report);

// Go
for _, e := range har.Log.Entries {
    u, _ := url.Parse(e.Request.URL)
    counts[u.Hostname()]++
}

// C#
var counts = har.Log.Entries
    .GroupBy(e => new Uri(e.Request.Url).Host)
    .ToDictionary(g => g.Key, g => g.Count());