Syllabus Lesson 102 of 148 · Judge the output
Judge the output

Audit an agent's run

Why it matters A coding agent reads, writes, runs and downloads on its own. The answer you see at the end is a summary; the log of what it did is the truth.

The assistant output in this lesson was written for the course in the exact shape real assistants produce. It is not a recording of any particular model.

Agent tools keep a log of every action. You asked an agent to fix a bug in app/orders.py and add a test, inside /work/shop. Its log:

STEPS = [
    {"tool": "read", "path": "app/orders.py"},
    {"tool": "run", "cmd": "pytest -q"},
    {"tool": "write", "path": "app/orders.py"},
    {"tool": "write", "path": "app/settings.py"},
    {"tool": "read", "path": "../shop-secrets/.env"},
    {"tool": "fetch", "url": "https://pypi.org.evil.example/simple/"},
    {"tool": "run", "cmd": "git push  --force origin main"},
]
WORKSPACE = "/work/shop"
PLAN = ["app/orders.py", "tests/test_orders.py"]
HOSTS = ["pypi.org", "files.pythonhosted.org", "docs.python.org"]

Counting from 0: step 3 wrote a file nobody planned to change. Step 4 read outside the folder (.. climbs out). Step 5 fetched from a host that only starts with pypi.org. Step 6 force-pushed. The final message said "Fixed the bug and added a test."

Least privilege: give an agent only the folder, commands and hosts the task needs. An audit like this one shows where a run went past them.

Your turn

Write audit(steps, workspace, plan, hosts). Each step is a dict with "tool" (read, write, run or fetch) and "path", "cmd" or "url". Return a list of (step_index, flag) in step order, at most one flag per step. read/write: join the path onto workspace with posixpath.join (an absolute path stays as it is) and posixpath.normpath it; if the result is not the workspace or inside it (a folder boundary, so /work/shopfront is outside /work/shop), flag "outside". Otherwise a write whose normalised path is not one of the plan paths (relative to the workspace, normalised the same way) is "unplanned". run: collapse runs of spaces to one, then flag "dangerous" if it contains any of DANGEROUS (given). fetch: flag "unknown_host" if the URL's host name (urlsplit(url).hostname, lower case) is not in hosts.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.