Syllabus Lesson 37 of 156 · HTTP from the inside
HTTP from the inside

URLs and link cleaning

Why it matters When cookies are blocked or partitioned, trackers carry your ID in the link itself, and every time you copy and share such a link you pass along the click ID with it. A link cleaner removes that decoration on every navigation without breaking the page. Browsers and blockers ship one; here you write yours, and the blocker module reuses the idea.

Think of a parcel label. The address is what the courier needs. Stickers added by everyone who handled it along the way ("came via the newsletter", "customer number 7f3a") are not. Cleaning a link means peeling off the stickers while leaving the address exactly as it was.

URL: a web address, in parts: https://shopa.example:443/p/tent?colour=green#reviews has a scheme (https), a host (shopa.example), a port (443), a path (/p/tent), a query (colour=green) and a fragment (reviews, which stays in the browser and is not sent to the server).

Query parameter: one name=value piece of the query, separated from the others by &.

Link decoration: parameters added to a link to carry an ID or campaign label from one site to another, such as utm_source, fbclid or gclid.

Two jobs: normalise, then strip

Normalise means writing the same address the same way every time. Scheme and host are not case sensitive, so HTTPS://SHOPA.Example becomes https://shopa.example. Port 443 is the default for HTTPS (80 for HTTP), so writing it adds nothing. An empty path is /. The path itself is case sensitive on many servers, so leave it alone.

Strip means removing tracking parameters only. STRIP_PARAMS in watched.leg2c_http lists the click-ID and campaign names (from ad platforms, email tools and social sites), and every name starting with utm_ goes too. Compare names in lower case and after decoding (utm%5Fsource is utm_source in disguise). Everything else stays exactly as written: q=red+tent must not become q=red%20tent, and order matters to some sites.

Percent-encoding: writing a character as % plus two hex digits, so it can travel safely in a URL: %5F is _, %26 is &. urllib.parse.unquote turns it back.

from urllib.parse import urlsplit
parts = urlsplit("https://Shop.Example:443/p?utm_source=x&id=5#top")
parts.scheme, parts.hostname, parts.port, parts.path, parts.query, parts.fragment
# ('https', 'shop.example', 443, '/p', 'utm_source=x&id=5', 'top')

Why not rebuild the query with urlencode? Because it re-encodes values and can quietly change them. Splitting the raw query on & and keeping the untouched pieces is safer. A nested address such as ?to=https%3A%2F%2F...%3Futm_source%3Dx is one encoded value of a normal parameter; leave it alone here (unwrapping bounce links is a job for the next lesson and the blocker).

A common misconception

"Strip every parameter to be safe." That breaks search results (q=), page numbers (page=2) and, worst of all, password-reset and login links (token=). One of the 40 graded cases is exactly that reset link. Cleaners in browsers use a list for the same reason (checked October 2026).

Try it Copy a link from a newsletter email or a social media post into a text editor. Can you spot utm_ parameters or a long click ID? Remove them, open the shorter link, and check the page still works.

In other languages

JavaScript:

function cleanLink(link) {
  const u = new URL(link);                 // lower-cases host, drops default port
  if (!["http:", "https:"].includes(u.protocol)) return link;
  const kept = u.search.slice(1).split("&").filter(piece => {
    const name = decodeURIComponent(piece.split("=")[0]).toLowerCase();
    return piece && !name.startsWith("utm_") && !STRIP.has(name);
  });
  u.search = kept.length ? "?" + kept.join("&") : "";
  return u.toString();
}

C#:

var u = new Uri(link);
var kept = u.Query.TrimStart('?').Split('&')
    .Where(p => p != "" && !IsTracking(Uri.UnescapeDataString(p.Split('=')[0])));
var b = new UriBuilder(u) { Query = string.Join("&", kept) };

Recap

  • Normalise the parts that are not case sensitive (scheme, host), drop default ports, and turn an empty path into /.
  • Strip only known tracking parameters, compared in lower case after decoding; keep every other piece exactly as written and in order.
  • Keep the fragment, leave non-web links alone, and never strip tokens a page needs.
Your turn

Write is_tracking_param(name): True when the name, percent-decoded with unquote and lower-cased, starts with utm_ or is in STRIP_PARAMS. Then write clean_link(url): leave non-http/https links unchanged; lower-case scheme and host; drop a default port; turn an empty path into /; split the query on &, drop empty pieces and tracking parameters, keep the rest exactly as written and in order; no ? when nothing is left; keep the #fragment. It is graded on the 40 cases in LINK_CASES plus random links.

Spotted a problem in this lesson? Report it

Code · runs in your browser
Output

        

      

      
      

        Back