Syllabus Lesson 247 of 250 · Working with Model Replies
Working with Model Replies

Cut Off, Refused, or Out of Scope

Why it matters A support bot that shows a user half a sentence, or an empty bubble, looks broken. Before you show any reply, decide what kind of reply it is.

The app here is a bank assistant whose system prompt says: answer only questions about accounts and cards, and for anything else reply with exactly OUT_OF_SCOPE. Five replies:

  • a normal answer;
  • OUT_OF_SCOPE exactly as asked;
  • the same signal written loosely (Out_of_scope. with a newline): models follow "reply exactly" almost always, so match it with some tolerance (ignore letter case, spaces and a trailing full stop);
  • an answer cut off at the token limit;
  • a refusal: stop_reason is "refusal", the content is empty, and a stop_details dict may say why.

Each kind needs a different screen: show the answer, show "I can only help with banking", ask the model to continue or retry with a higher limit, or show a polite decline.

About these replies: they are simulated. Each one was written for this course in exactly the shape the Claude Messages API returns (content blocks, stop_reason, usage), and each copies something real models do. They are not recordings of a live model, so the exact wording is ours; the shapes and the problems are the real ones your code will meet.

Your turn

Write classify_reply(reply) returning one of "refused" (stop_reason is "refusal"), "cut_off" (stop_reason is "max_tokens"), "out_of_scope" (the text, with surrounding spaces and a trailing . removed, equals OUT_OF_SCOPE ignoring case), "empty" (no text at all), or "answer". Check them in that order. Keep the starter's REPLIES as it is: the check reads it.

Spotted a problem in this lesson? Report it · Ask about this lesson

This exercise opens soon

Finish the lesson before it and this one unlocks. The whole course is free, no account needed, and you can read this lesson right now.