Dictionaries and counting
Why it matters Before a language model sees text, someone counts its words or pieces. A dictionary is the tool for counting.
A dictionary maps a key to a value, like a word to how often it appears: {'the': 3, 'cat': 1}. To count, look up the current count (0 if new) and add 1.
counts = {}
for w in words:
counts[w] = counts.get(w, 0) + 1
counts.get(w, 0) means: the count for w if we have one, else 0.
Tiny numbers
"The cat. The hat!" has words the, cat, the, hat once you lower-case it and strip the punctuation, so the counts are the: 2, cat: 1, hat: 1. Without cleaning, "The" and "the" and "cat." would count as different words.
Recap Clean, split, count with get(key, 0) + 1.
Words and symbols in this lesson
- mean, spread
- the average, and roughly how far values sit from it (standard deviation)
Write word_counts(text) that returns a dictionary of word counts. Lower-case the text, and remove these characters, then split on any whitespace (spaces, tabs, new lines: text.split() with no argument does that): . , ! ? ; : " '. Empty text gives an empty dictionary.
Spotted a problem in this lesson? Report it · Ask about this lesson
This exercise opens soon
Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.
This browser is blocking cookies for learn.floati.life, so lessons you finished here cannot open in order. Allow cookies for this site and reload.