To test an AI agent skill before keeping it, give it a handful of recent tasks it wasn't built from, run those tasks once with the skill and once without it, and grade both runs against checks you wrote before the first run. Then confirm it switches on only when it should, count how often you still correct it, and decide to keep it, change it and rerun the same tasks, or remove it. How the skill felt to use is the weakest evidence on that list.
Your sense that a tool helped is weak evidence
METR, an AI evaluation research group, published a randomized trial of experienced open-source developers on July 10, 2025. In that July 2025 METR study, 16 developers brought 246 real issues from their own repositories, and each issue was randomly assigned to allow or disallow AI tools. When AI was allowed, the developers took 19% longer to finish, METR reported in July 2025.
For anyone testing a skill, the useful finding is what those developers believed. They expected AI to speed them up by 24% before the work, and they still believed it had sped them up by 20% after finishing, according to the same July 2025 METR write-up.
Use that study for its method. It says nothing reliable about today's tools: METR's page now marks those results as out of date, and on February 24, 2026 METR wrote that developers are likely more sped up by AI tools in early 2026 than its early-2025 estimates, while calling its own newer data only very weak evidence of how large that change is, because of selection effects in the experiment.
The method holds up. Comparable work is done with and without the tool, and the result is read from a record instead of a memory. A skill you just built deserves the same treatment, because you're the person most likely to overrate it.
The 5 checks before you keep a skill
Write the checks before the first run. OpenAI's January 22, 2026 guide to testing agent skills says to define success before you write the skill, and it splits checks into outcome, process, style and efficiency goals. For a founder those can be a few plain sentences: the output is something you'd send without edits, it used the right data, it followed your format, and it got there without several retries. Write them down first so a good-looking result can't move the bar.
Use work the skill hasn't seen. Pick recent tasks it wasn't built from, because a skill can look perfect on the examples it learned from. For the product-sense round of my agent's MBA, I had it go back through 564 corrections I'd made over 3 months, which grouped into 14 patterns, as I wrote in I Gave My Agent An MBA on November 1, 2026. In that same essay I noted it had just built me a pretty bad survey flow. New work is where a skill shows what it actually learned.
Run the same tasks without the skill. Give the agent the identical tasks with the skill turned off. Start both runs from the same setup: a fresh session, a separate copy of the same files, the same saved memory and the same tool access, with the model and instructions unchanged apart from the skill. A new chat alone can still carry over files or memory the last run changed. Then grade both outputs side by side against the checks from step 1. Agents don't give the same answer twice, so run any task with a close result again on both sides. That comparison is the METR design in miniature. If you can, hide which output came from which run until you've graded both.
Check that it switches on at the right times. OpenAI's January 2026 guide says a skill's name and description are the primary signals Codex uses to decide whether to invoke the skill at all. Add a few tasks where the skill should stay off, and confirm it does. A skill that fires on the wrong job can quietly change work you never asked it to touch.
Count your corrections, and check that old fixes hold. In 5 AI-Native Metrics to Track on August 4, 2026, I described intervention rate: the percentage of agent outputs that need a human correction. Track it for the tasks in this test. My November 2026 MBA essay also says that when I correct the agent now, the fix goes into the skill and it checks against it next time, so rerun a task that caused an earlier correction and confirm the same mistake doesn't come back.
If you build skills with the free AI MBA skill, its third step is putting the new skill to the test on real work, and the download includes a guide to checking the result. These 5 checks are a way to run that step.
Keep, change or remove
The test ends in one of 3 calls on the skill, and a split result counts as change, not keep:
- Keep it when the skill run beats the no-skill run on most of the new tasks against your written checks, it stays off for tasks outside its job, and old corrections hold.
- Change it when it wins some tasks and fails a check you can name. Edit the skill for that failure, then rerun the same tasks so you're comparing like with like.
- Remove it when it still can't beat the no-skill run after 1 round of changes, or when you can't say what it does better. That cutoff is my own rule of thumb, not a standard. A skill that adds nothing is still one more set of instructions the agent has to weigh.
This is the same loop good delegation runs on. In The Delegation Flywheel on January 21, 2026, I shared Aaron Hayslip's framework for assistants: write a doc for the task, the assistant runs it, you review the outcome and give feedback, and the assistant updates the doc. A skill is that doc, and this test is the review step.
A keep, change or remove card to copy
Copy this into the skill's notes and fill it in each time you test:
- Skill name and the single job it's for.
- Checks written before the first run, each one something you can mark pass or fail.
- New tasks used for the test, with the date of each.
- Starting setup for both runs: model, instructions, files, saved memory and tool access, matched except for the skill.
- Result with the skill and without it, task by task, graded against the checks.
- Tasks where the skill should have stayed off, and whether it did.
- Corrections made during the test, and which earlier fix you retested.
- Decision: keep, change or remove, with 1 sentence on why.
- Tripwire: IF the skill hasn't beaten the no-skill run on new tasks BY the end of the second test round, THEN remove it and keep the notes.
Run the card again when you switch the model underneath the agent. A skill that won last month was tested against last month's agent.
Where this test doesn't apply
A test on a handful of tasks is a first screen. It tells you which version did better on your own work this time, which is enough to make a reversible call on a skill you can always change back. It won't give you a statistic you can publish, and it can't detect a small difference between the 2 runs. When the results split, call it inconclusive and add tasks before you decide.
Skip the formal test for a skill you'll use once, or one that only formats text you'll read anyway. Test harder, with more tasks and someone else grading, when a skill touches customer data, money or anything that goes to a customer without review.
The setup you run doesn't change this. Whether it's one company brain or a crew of named agents, which I compared in You Don't Need AI Teammates in August 2026, each skill still has to earn its place against a baseline. And keep reading the outputs yourself. In The 5 Levels of Work in October 2025, I wrote that hiring people who solve problems on their own doesn't excuse you from knowing what's going on in your business, and that goes for agents too.
Think AI's March 2026 video covers the testing features Anthropic added to its skill-creator tool, including evals, benchmarks and side-by-side output comparisons, according to the video's YouTube description:
Frequently asked questions
How do you test an AI agent skill before keeping it?
Run a handful of recent tasks the skill wasn't built from, once with the skill and once without it, and grade both against checks you wrote first. Also confirm it stays off for tasks outside its job and that earlier corrections don't come back.
How do I test my AI agent?
Give it real tasks with checks written in advance, and compare its output with a baseline: the same tasks done without the agent, or without the change you're testing. Grade both against the checks, and track how often you still have to correct it.
What are AI evals?
OpenAI's January 22, 2026 guide to testing agent skills describes an eval as a prompt, a captured run, a small set of checks and a score you can compare over time. For a founder, a short list of real tasks with written pass or fail checks is a usable first eval.
Write the checks before you run your newest skill again. If you're building the next one, get the free AI MBA skill, and for one essay every Sunday on building leverage in your life and work, get the Sunday issue.
Get the next one.
One essay every Sunday on how to build leverage in your life and work.