Syllabus Lesson 234 of 246 · Module 22: Judging claims
Module 22: Judging claims

Replicate it, then change one thing

Why it matters A result you can rerun is worth more than any number of results you have to take on trust. And a result that holds only for the exact settings it was found with is a narrower claim than its headline.

In Module 21 you saw: with 5 features that are each on 3% of the time, a model of 2 numbers keeps all 5 (sparsity 0.97, importance 0.9, seed 0). That was one run.

Replicate it: the same settings with seeds 1, 2 and 3 (1,500 steps of 256 examples each, enough for this toy: the counts match the longer runs). Then change one thing the claim did not mention: how quickly importance falls from feature to feature, 0.9, 0.7 and 0.5 (at 0.5 the last feature counts 1/16 as much as the first).

Predict first: at importance 0.5, will it still keep all 5?

Words and symbols in this lesson
seed
the number that fixes a run's random choices; another seed gives another result
Your turn

Set my_guess to "yes" or "no". Fill kept[importance] with the number of stored features for seeds 1, 2, 3 (a list of three) at importance 0.9, 0.7 and 0.5. Set holds_at to the importances where all three seeds keep all 5.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.