LESSON 5 OF 6
Test it before you trust it
BY THE END OF THIS LESSON
Prove an automation works on the inputs it will actually meet.
The happy path proves almost nothing
Everyone tests with a clean example they wrote themselves, and it passes. That test tells you the steps are connected, which is worth knowing and is not the same as working.
Real inputs are messier than anything you would invent: blank fields, someone replying in a different language, an attachment where you expected text, the same thing submitted twice.
Test with your own history
- The last twenty real ones
- Run those through
- Read every result
You already have the best test set there is: the last twenty real ones. Take them from your inbox, your form responses, your ticket queue.
Run those through and read every result. Twenty is small enough to check by hand and large enough that the awkward cases are in there — because they are always in there.
Then run it beside the manual process
Before you compare the two
Run it beside the person for a week
Before switching over, run the automation for a week while the person keeps doing the job. Compare the two.
It is the only way to learn what it gets wrong at a rate that matters. A disagreement in one case out of twenty is invisible in testing and obvious in a week of real traffic — and you find it while somebody is still checking.
WORKED EXAMPLE
TESTED ONCE
Submitted a test form with my own name and a sensible message. It worked, so I turned it on.
TESTED ON HISTORY, THEN IN PARALLEL
Ran the last twenty real submissions through it and read all twenty results. Three were classified differently from how I would have done it, and two of those were genuinely ambiguous — so I added the "unclear" route. Then left it running beside the manual process for a week and compared.
The single clean test only proved the steps were wired together. The twenty real ones surfaced a category that did not exist, and the parallel week is what shows the error rate on traffic you did not choose.
YOUR TURN
Build the test set from what you already have.
RUN THIS
I am about to turn on this automation: [describe it]. Tell me exactly where to find twenty real past inputs for it, and what to look for in the results — including the awkward shapes I should specifically check for, like blanks, duplicates and inputs in another language. Then describe how to run it in parallel for a week and what I should compare.
What a good result looks like
The awkward shapes should be specific to your data, not generic. If the twenty run through with zero disagreements, be suspicious — check whether the automation is actually reading the input.
KNOWLEDGE CHECK
Answer all 2 correctly to complete this lesson.
1. What does a single clean test actually prove?
2. Why run the automation beside the manual process for a week?
REMEMBER THIS
Test on twenty real past inputs, then run beside the person for a week before switching over.