Syllabus Lesson 97 of 148 · Judge the output
Judge the output

Read the output, not the tone

Why it matters An assistant's answer is a bundle of claims: these packages exist, this diff only fixes the bug, this code handles the edge case, these commands are safe. Each claim can be checked, and most checks are cheap.

Earlier modules proved advice wrong with a failing test. This module goes one step wider: it checks the parts of an answer people skip because they look like plumbing.

  • Names: packages, functions and flags. In a large study of code-generating models, on average at least 5.2% of the packages that commercial models recommended, and 21.7% of those from open models, did not exist, with over 200,000 distinct invented names (Spracklen et al., USENIX Security 2025). An attacker can register an invented name and wait.
  • Scope: a diff that fixes the bug and quietly changes three other functions.
  • Choices: a vague request makes the assistant decide things you never said, such as letter case, order or empty values.
  • Agreement: asking twice and getting the same answer twice does not make it right.
  • Actions: an agent that runs commands can step outside the folder you meant.

Slopsquatting: registering a package under a name that models tend to invent, so code that installs it runs the attacker's code.

Typosquatting: the older trick of registering a name one slip away from a popular package (requets), waiting for a typing mistake, by a person or a model.

The assistant output in this module was written for the course in the exact shape real assistants produce. It is not a recording of any particular model.

Recap Split an answer into claims and check the cheap ones every time: names, scope, choices, agreement, actions.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.