Syllabus Lesson 101 of 148 · Judge the output
Judge the output

Six answers to one question

Why it matters Asking again and taking the most common answer feels like a second opinion. Samples from one model share its habits, so the majority can be wrong together.

The assistant output in this lesson was written for the course in the exact shape real assistants produce. It is not a recording of any particular model.

A teammate asked six times for "a function that returns the median of a list of numbers". Four of the six answers pick one middle element after sorting, which is wrong for lists of even length (the median of [1, 2, 3, 4] is 2.5); one of those also sorts the caller's list in place. Two are right. Vote on what they return, and a wrong answer wins three to two.

Running them is cheaper than arguing about them. A few cases with known answers, including the one the common answer misses, sort the six at once.

load_sample runs a sample's code here, in this browser tab, like your own. These samples were written for the course. With real assistant output, read it first and run it somewhere disposable, such as a container or a throwaway virtual machine, never next to your keys and data.

Self-consistency: sampling several answers and taking the majority. It helps when the errors are random; it does not help when the model has one systematic blind spot.

Your turn

Write rank(samples, name, cases). samples is a list of code strings, each meant to define a function called name. cases is a list of (args, expected) pairs, args being a tuple. For each sample, get its function with load_sample(code, name) from creative.samples; if that gives None (the code failed to run or defines no name), every case fails. Otherwise call the function for each case (pass a copy of any list argument, so one sample cannot change the case for the next); a case passes when the call returns a value equal to expected, and fails if it differs or raises. Return one dict per sample, {"sample": i, "passed": n, "failed": [case indexes]}, most passes first, ties by sample index.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.