Syllabus Lesson 149 of 156 · Measure the web like a researcher
Measure the web like a researcher

Honest statistics

Why it matters A measurement is only useful if readers know how far to trust it. Writing down how your sample was chosen, how uncertain the number is, and what you could not see is what separates a study from a guess. It also protects you from fooling yourself.

Imagine asking people at a gym how often they exercise, then announcing "the average person works out five times a week". The arithmetic may be perfect. The problem is who you asked.

Sample: the part of the world you actually measured, such as the 1,000 sites you crawled.

Population: the whole thing you want to say something about, such as "websites people use".

Sampling bias: a difference between your sample and the population that pushes the result one way, however large the sample.

Where web crawls are biased

  • Popular sites only. Rankings favour big sites. Small sites, which most of the web is, may behave differently.
  • Regions and languages. A crawler in one country sees that country's versions of sites, often in one language.
  • Devices. Desktop crawls miss what mobile sites and apps do, and much tracking now happens in apps.
  • Logged-out pages. Crawlers rarely log in, so account pages, checkouts and feeds go unmeasured.
  • Bot detection. Some sites behave differently for automated browsers.
  • The labels. Your tracker count is really "sites the blocklist calls trackers". A list that misses trackers lowers your number.

How uncertain is a share?

Even without bias, a sample wobbles by chance. For a share p measured on n sites, a common rough 95% margin of error is 1.96 * sqrt(p * (1 - p) / n).

import math
p, n = 0.6, 1000          # 600 of 1,000 sites had a tracker
margin = 1.96 * math.sqrt(p * (1 - p) / n)
print(round(100 * p), "% plus or minus", round(100 * margin, 1), "%")   # 60 % plus or minus 3.0 %

This margin covers chance only. It says nothing about bias, which is often the bigger error, so you still need the list above.

Writing up limits

A good limits section is short, specific and honest. For each limit say what it is, which way it probably pushes the result, and what would fix it. For example:

  • "We visited home pages only. Inner pages often load more trackers, so our counts are probably low."
  • "We used one blocklist for labels. Trackers it misses are not counted."
  • "The crawl ran on one day, from one location, on desktop."

A common misconception: "Listing limits makes a study look weak." It does the opposite. Readers trust a result more when they can see its edges, and limits tell the next researcher what to measure.

Two more habits: give the date and the exact method so someone can repeat it, and avoid turning "we saw it on 60% of sites" into "60% of people are tracked". Sites are not people.

Try it Take your crawl from lesson 146. Compute the margin of error for its tracker share, then write three limits for it, each saying which way it pushes the number.

Recap

  • Sampling bias comes from who or what you measured, and a bigger sample does not cure it.
  • Crawls miss small sites, other regions, mobile and apps, logged-in pages, and whatever the labels miss.
  • A margin of error such as 1.96 * sqrt(p(1 - p)/n) covers chance only.
  • Write limits that say what, which direction, and what would fix it.

Spotted a problem in this lesson? Report it

Code · runs in your browser
Output

        

      

      
      

        Back