On this page
A thumbnail decision log records, for every cover decision, what you expected, what you changed, what happened and what you will do differently next time, in a form the next brief can use. A spreadsheet of click-through rates is a different document, and a much less useful one. The A/B testing guide ends on the rule "write the lesson, not the score"; this is the template that makes the rule easy to follow, a grammar for the lesson itself, worked examples of entries that hold up months later, and the quarterly review that turns a page of lessons into a short list of house rules.
Why a lesson compounds and a score does not
A score is a number tied to the conditions that produced it. A CTR of any size belongs to one traffic mix, one video age, one week of the calendar and one row of neighbours, and none of those will be the same next month. You cannot compare it with the next score without rebuilding all four conditions, which is why a baseline has to be split by source and age before any single figure means anything. A score answers "how did that cover do" and the answer is already out of date.
A lesson is a statement about the viewer. "On reviews, doubt beats shock" does not need the conditions rebuilt; it is applied by choosing doubt on the next review. Two lessons about the same lever either agree, and you have a rule, or disagree, and you have a condition to find. That is what compounding means here: each entry makes the next brief faster, and the log gets more useful as it gets longer, where a list of scores gets longer and no more useful.
Our view is that a channel's log is worth more than its analytics history, because the analytics describe what happened and the log describes what the channel learned. Two people with the same analytics and different logs will make different covers, and the one with the better log will make them faster.
The template
Ten fields, one row per decision. The width is deliberate: a narrower log loses the fields that make a lesson reusable, and a wider one does not get filled in.
| Field | What goes in it | Example |
|---|---|---|
| Date and video | When, and which video by working title | 2026-09, "Air fryer, one month" |
| Format | The video type the lesson applies to | Review |
| Decision type | Test, manual swap, refresh, judgement call, or not tested | Test |
| Hypothesis | Format, lever, expected direction, viewer reason, written before the result | On reviews, doubt beats delight because the viewer wants a verdict |
| Variants | What changed and what stayed, both halves if a title was involved | A doubt face, B delight face; same headline, layout, colour, title |
| Method | Test & Compare, alternating swap, concept run, or none | Test & Compare, two slots |
| Result | The result state, the manual gap, or "shipped A" | Winner: A |
| Lesson | One sentence in the grammar below | Doubt beats delight on reviews; the viewer wants a verdict, not a fan |
| Confidence | Hint, lean, rule, or retired | Lean (first Winner) |
| Next use | The specific next decision this changes | Doubt on the next two reviews; retest against calm after that |
A blank row to copy:
| Date and video | Format | Decision type | Hypothesis | Variants | Method | Result | Lesson | Confidence | Next use |
|---|---|---|---|---|---|---|---|---|---|
Two fields do most of the work. Hypothesis is written before the test, which is the only thing that stops the entry being a story told in hindsight; the variant ladder gives the sentence shape for each kind of test. Next use turns the lesson into an instruction with a video attached, which is the difference between a log that is read and a log that is admired. An entry with an empty next-use field has not finished.
How to write the lesson: a grammar
A lesson has five parts, in this order: the lever's winning side, the losing side, the format, the condition if there is one, and the confidence. "Doubt beat shock on reviews when the product was expensive; Winner." Every part narrows where the lesson applies, and a lesson that leaves out the format or the condition will be applied where it is false.
The table below rewrites the entries we see most often. The left column is what gets written; the right is the same result in the grammar.
| Written as a score | Written as a lesson |
|---|---|
| "B won" | "The number headline beat the word headline on finance stories; the picture could not show the stake; Winner." |
| "New style did better" | "Object-led beat face-led on unboxings; the viewer was checking the product; Preferred, retest once." |
| "No difference" | "Expression did not move tutorials; calm and shock performed the same; stop testing emotion on this format." |
| "CTR went up after the swap" | "Manual swap on an evergreen how-to: the specific title beat the general one by a wide gap across both pairs of windows; the week held no holiday; lean." |
| "Kept the old one" | "Not tested: video under a week old and impressions still climbing; judged too early to swap; revisit at day 28." |
| "Package B won" | "The method promise beat the stakes promise on debt stories; two packages, so the half is unknown; wedge next time." |
Some habits keep entries honest. Write "Performed the same" entries as findings, because a lever that does not matter is a lever you can stop spending time on. Write manual swaps with the calendar named, because a swap that straddled a seasonal edge is not a lesson about the cover. And write package tests as lessons about the promise, not the picture; how a title-and-cover test is read sets out why crediting the cover after a package win is the most common false entry.
Take a hypothetical entry from start to finish. The video is a budgeting explainer. Variant A is a calm face on a dark green ground with $0 LEFT in yellow at a third of the height; variant B is the same face, ground and layout with MY BUDGET in the same yellow. The hypothesis, written before the test: on budgeting explainers, a stake headline beats a subject headline because the viewer wants a consequence, not a topic. Test & Compare returns Winner: A. The score entry reads "A won", which tells the next brief nothing. The lesson entry reads: "The stake headline beat the subject headline on budgeting explainers; the viewer wanted a consequence; Lean." Next use: stake headline on the next two budgeting videos, then retest against a number-plus-noun headline. The two entries describe the same result. Only one of them can be handed to someone else.
Write the lesson for the person briefing a cover in six months, who may not be you and will not remember the test. If the sentence needs the video open to make sense, it is not finished. If it could be pasted into a brief as an instruction, it is.
The confidence ladder
Confidence is a field, not a feeling, and it moves only when evidence arrives. Four levels are enough.
| Level | What earns it | What it permits |
|---|---|---|
| Hint | One manual gap, one Preferred, or one concept run that leaned one way | Try the direction on the next video; do not brief it as a rule |
| Lean | One Winner, or two hints in the same direction on the same format | Use it by default on that format; keep one retest scheduled |
| Rule | The lean repeated on the same format, or a Winner plus an agreeing hint | Brief it without testing; move testing to the next lever |
| Retired | A rule that stopped holding, or a lever the format no longer varies | Note why; keep the entry so nobody re-runs the old test |
A lesson climbs one level per agreeing result and drops one per disagreeing result. It never jumps to rule on a single test however clear the Winner, because one video's audience is one audience; the small-channel method of running a concept across several uploads is the fastest way to earn the repeat. And a rule is not permanent. Audiences habituate, the niche converges, and a rule that was true for a year can retire; the quarterly review below is where that happens.
Log the decisions that were not tests
Most cover decisions are never tested, and a log that only holds tests hides where a channel's habits live. Every upload is a decision. So is every swap on a live video, every refresh of an old cover, and every time you chose not to test. Give each of them a row with the decision type filled in and the hypothesis written as the reason.
- Judgement call: the cover was shipped without a test. Hypothesis is why this cover; next use is what would make you test it.
- Manual swap: the windows, the source, the gap, and the calendar note. Whether the swap should have happened at all is decided before the row, not by it.
- Refresh: which old cover, what changed, and the source-matched comparison a month later.
- Not tested: the reason. Too young, too few impressions, ineligible, or no hypothesis. The last one is the useful entry, because a run of "no hypothesis" rows on one format tells you where the next test belongs.
We would rather have a log with thirty honest "not tested: no hypothesis" rows than one with ten tidy test entries, even though the first looks like a channel that never tests and the second looks like rigour. The thirty rows show which format runs on habit. The ten hide it.
When channels start logging untested decisions, the same discovery comes up: the format they upload most is the one they test least, because its covers are made on habit. The log makes the habit visible, and the first test on that format is usually the most valuable one the channel runs that year.
The quarterly review
Once a quarter, read the log as a whole rather than a row at a time. It takes an hour and it is where lessons become rules. The weekly review feeds this one; the weekly loop writes rows, the quarterly loop reads them.
- Gather. Every row from the quarter, including the untested ones.
- Group by lever and format. All the expression rows for tutorials together, all the headline rows for reviews together. Patterns hide across rows and appear in groups.
- Move confidence. Promote lessons that repeated, demote lessons that were contradicted, retire rules that stopped holding. Write the date beside each change.
- Find the gaps. Which formats have only judgement rows? Which levers have never been tested on your biggest format? Those are the next quarter's hypotheses, and they go into the log now as planned tests.
- Write the house rules page. One line per rule, with the format, the date it was promoted and the lever it settles. This page, not the log, is what the brief reads. It should fit on one screen, and if it grows past that, some rules are conditions of other rules and should be merged.
Rebuild the CTR baseline in the same sitting, because the channel's mix drifts over a quarter and the baseline is what every manual entry in the next quarter is read against.
Retire rules on purpose, not by forgetting them. A rule that has not been retested in a year is a belief. Put one retest of your oldest rule into each quarter's plan; if it holds, the date moves and the rule stays. If it does not, you have caught habituation before it cost a quarter of covers.
Keeping the log alive
A log dies when it becomes a chore or when nobody reads it. These keep it in use.
- One row per decision, written the day the result arrives. Not the day the test starts, when the hypothesis is written, and not a week later, when the reason has faded.
- Read it at brief time. The research step of the five-step thumbnail workflow starts with the house rules page and the log's rows for this format, so the brief begins from what the channel already knows.
- Keep the runner-up. Every test's losing variant is the refresh candidate for that video, and the next-use field can point to it by name. On the bench the history strip and versions keep every render, so "next use: the calm-face version from the history" is an instruction someone can follow in a minute rather than a rebuild. Tweak makes the retest variant from the winning render without touching anything but the lever.
- One owner. On a channel with an editor or an agency, the log has a single person who writes rows and runs the review, and everyone reads the house rules page.
What to do next
- Copy the blank row into whatever you already use for planning, and fill in the last three cover decisions from memory, untested ones included. Notice which fields you cannot fill.
- Write the hypothesis for the next upload before the cover is made, in the grammar, with the "because".
- Put the first quarterly review in the calendar now, and plan one retest of whatever you currently believe most firmly about your covers.