LESSON 4 OF 6
What happens when it breaks
BY THE END OF THIS LESSON
Design the failure path before the automation runs unattended.
It will break, and you will not be told
A service will be briefly unavailable. Somebody will rename a column. An input will arrive in a shape you never saw. None of these are unusual, and all of them happen at a time you are not looking.
The question is not whether it breaks. It is whether you find out from your own alert or from the person who was expecting the output.
Three failures, three different answers
A step errored. Retry it a couple of times — many are momentary — and if it still fails, stop and tell someone. Retrying forever hides the problem.
A step returned something unusable. The model gave a paragraph where a category was wanted. Route it to a person rather than passing it on.
Nothing happened at all. The trigger stopped firing. This is the one nobody catches, because silence looks exactly like a quiet week.
Watch for silence
- It stopped three weeks ago
- Looks identical to nothing to do
- Something that says so
The failure that costs most is the one that produces nothing. An automation that stopped three weeks ago looks identical to one that had nothing to do.
So add the check nobody thinks of: something that notices when the automation has not run in longer than it should have, and says so. It is the only failure you cannot detect by looking at the output, because there is no output.
WORKED EXAMPLE
NO FAILURE PATH
When a form is submitted, classify it and create a task. (No error handling configured.)
FAILURE PATH DESIGNED
On a step error: retry twice, then post to the ops channel with the input that failed. If the classification comes back as "unclear" or anything not in the list: create the task in a triage queue instead, marked for a person. If no form has been processed in seven days: post a message asking whether that is expected.
All three failures now surface somewhere a person looks, and each one carries the thing needed to diagnose it. The third catches the case where the automation quietly stopped, which is the one that otherwise runs for a month.
YOUR TURN
Design the failure path for something you already run.
RUN THIS
Here is an automation I have running: [describe the steps]. For each step, tell me what happens today if it fails, and who would find out. Then write the three failure paths: a step erroring, a step returning something unusable, and the automation not running at all. Say exactly where each alert should go and what information it needs to carry.
What a good result looks like
If the honest answer to "who would find out" is "nobody, until someone complains", that is the finding. The alert should carry the failing input, not just the word "failed".
KNOWLEDGE CHECK
Answer all 2 correctly to complete this lesson.
1. Why is "nothing happened at all" the most dangerous failure?
2. Why not simply retry a failing step until it succeeds?
REMEMBER THIS
Design three failure paths — errored, unusable, and silent — before it runs unattended.