Audit a practice app

The practice capture

Our practice app is FitFox, a made-up step counter whose own site is fitfox.example. Its capture is a Python list. Each item is one request, as an interception proxy would log it:

from watched.leg5b_apps import fitness_app_log

log = fitness_app_log()
print(len(log))      # 12
print(log[3])
# {'t': 3, 'device': 'phone', 'method': 'GET',
#  'url': 'https://sdk.adco.example/ad?idfa=00000000-0000-0000-0000-000000000000&slot=banner&lat=53.80&lon=-1.55',
#  'headers': {}, 'body': {}}

t is seconds since the app opened. headers is a dict of request headers. body is the data the app sent, already decoded into a dict (it is empty for GET requests, which carry their data in the address).

Three places data hides

An app can put a piece of data in three places, like writing on the envelope, on a sticky note on the envelope, or in the letter:

1. The query string: the part of the URL after ?, as name=value pairs joined by &.
2. Headers: extra labels on the request, such as X-Device-Id: D-7F3A9C.
3. The body: the main data of a POST request.

Python's standard library splits a query string for you. urlsplit breaks a URL into parts, and parse_qsl turns the query into a list of (name, value) pairs:

from urllib.parse import urlsplit, parse_qsl

parts = urlsplit("https://tiles.mapkit.example/v1/tile?lat=53.80&zoom=14")
print(parts.hostname)          # tiles.mapkit.example
print(parts.query)             # lat=53.80&zoom=14
print(parse_qsl(parts.query))  # [('lat', '53.80'), ('zoom', '14')]

Making names comparable

The same idea can be spelled device_id in a body and X-Device-Id in a header (the X- is an old habit for "custom header"). Before comparing, normalise every name: lower case, turn - into _, and drop a leading x_. The practice world gives you this as normalise_name:

from watched.leg5b_apps import normalise_name
print(normalise_name("X-Device-Id"))   # device_id

Values come in different types (the body has numbers like 8412). Turn every value into a string with str(value), so they all compare the same way.

What counts as what

The practice world also gives you two sets of normalised names. IDENTIFIER_NAMES holds names that carry an ID for you or your phone (device_id, user_id, idfa, email_sha256 and so on). LOCATION_NAMES holds names that say where you are (lat, lon, city, postcode and so on).

First party or not

A request is to an SDK when its host is on a different site from the app's own. site_of from watched gives the site: site_of("cdn.fitfox.example") is "fitfox.example", and site_of("sdk.adco.example") is "adco.example".

A common misconception

"Only requests to strange domains matter." FitFox sends your heart rate and exact position to its own server, and a hashed email to partners.fitfox.example. That last one looks first party, but the name alone cannot tell you who runs it. Your audit will not flag it; the next exercise will.

function allFields(entry) {
  const norm = n => n.toLowerCase().replaceAll("-", "_").replace(/^x_/, "");
  const out = [];
  for (const [k, v] of new URL(entry.url).searchParams) out.push([norm(k), v]);
  for (const [k, v] of Object.entries(entry.headers)) out.push([norm(k), String(v)]);
  for (const [k, v] of Object.entries(entry.body)) out.push([norm(k), String(v)]);
  return out;
}

u, _ := url.Parse(rawURL)
for name, values := range u.Query() {
    for _, v := range values { fmt.Println(name, v) }
}
fmt.Println(u.Hostname())