How to Audit Your AI Work: 4 of 67 Moved a Number
AI makes building cheap, so it is easy to stay busy and never check if any of it worked. One founder just audited his own output and the result was humbling. Here is the same audit, ready for your shop, clinic or agency.
Building with AI feels great. Something new ships every day.
But did any of it make you money? Here is a simple audit that answers that, and how to automate it.
What did the founder actually find?
Noah Kagan, the founder of AppSumo, posted a confession on X. He had been building with Claude like crazy: 738 pull requests since September.
So he asked Claude to grade the work. Out of 67 things they actually shipped, only 4 clearly moved a number. Only 8% of the work was even for AppSumo, the business that pays for everything.
Four wins out of 67 is the number to remember. It was his own work, and he was proud of it until he checked.
There was a second problem. He had sent Claude 2,045 messages. He asked "did that work?" only 25 times.
And two AppSumo tests had already found winners. They sat for two weeks because nobody made the call. As he put it, nobody was him.
Why does this happen to small businesses too?
You may not write code. But the pattern is the same.
AI tools make it cheap to produce things. A new chatbot flow. A new email sequence. A new report. A new Instagram caption template.
When building is nearly free, the scarce thing is deciding what matters. Kagan said it himself: he was doing whatever felt productive instead of what moved things forward.
Here is how it looks in a typical SME:
- A shop owner sets up an AI reply bot, then never checks if replies turn into orders.
- A clinic automates appointment reminders, then never compares no-show rates before and after.
- An agency builds five automations, but nobody knows which one saves hours.
The work is real. The proof is missing.
How do you run a "did it move a number" audit?
You can do this in one afternoon. You need a list of what you changed and a few numbers you already track.
Step 1: List everything you changed since a fixed date. Pick the start of the quarter or the month. Include tools, automations, new pages, new campaigns and new processes.
Step 2: Write the one number each change was meant to move. Orders per week. Enquiries answered within an hour. Hours spent on invoicing. No-shows. If you cannot name a number, that is already a finding.
Step 3: Check whether it moved. Compare the number before and after. Be honest if you never measured it.
Step 4: Sort each change into stop, keep or finish. Stop what did nothing. Keep what worked. Finish what is half-done or waiting for your decision.
Step 5: Clear the decision queue. Kagan's two winning tests waited two weeks for him. Look for your own waiting decisions and make them this week.
You can also hand the job to an AI assistant. Kagan shared the idea of his prompt. He asked Claude to look at everything shipped since a date and every message he sent. For each change, it should say what number it was meant to move, whether it moved, and what he never checked. Then it should rank what to stop, keep and finish.
You can adapt that to your own notes, task lists and chat history.
One new rule is worth copying: nothing goes live unless you know which number it should move.
What this means for businesses in Nepal
Most SMEs here have small teams and tight budgets. A wasted month hurts more.
So the rule matters even more for you. Before you pay for a tool or ask someone to build an automation, write down one sentence: "This should move [number] from X to Y."
Start with numbers you already have. Orders in your WhatsApp or Instagram inbox. Appointments in your diary. Invoices in a spreadsheet. You do not need a fancy dashboard.
If you cannot measure it today, the first job is to measure it, not to build more.
This fits our earlier advice on keeping a first AI project small. Pick one task, track the hours, then decide.
How to automate this with n8n
The audit works best as a weekly habit, not a one-off. n8n can run the boring part for you.
- Keep a simple change log. A Google Sheet with four columns: date, what changed, the number it should move, and the owner.
- Keep a metrics sheet. One row per week with your key numbers: orders, enquiries, hours spent, no-shows.
- Add a weekly schedule trigger in n8n. Run it every Monday morning.
- Read both sheets and send them to an AI model. Ask it to list which changes had a number that moved, which were never measured, and which decisions are waiting.
- Send the summary to email or Slack. Keep it to one page so you actually read it.
- Add a reminder for open decisions. If an item sits for more than a week, the workflow nudges the owner.
You do not have to start from zero. These community templates are close to this idea:
- n8n template: Send weekly narrative client reports with Google Sheets, Claude, and Gmail
- n8n template: Forecast sales trends and weekly reports with Stripe, Sheets, Slack, and Gemini AI
They are built for other reports, so you will need to change the data and the prompt. Test with a small sheet first.
FAQ
How do I know if an AI tool is actually working for my business?
Pick one number before you start, such as hours spent or orders per week. Measure it before and after. If it did not move, the tool is not working yet.
How often should I audit my automations?
A quick weekly check and a deeper monthly review work well. The weekly check can be automated. The monthly review needs your judgement.
Can I use ChatGPT or Claude to audit my own work?
Yes. Paste in your change list and your numbers, and ask what each change was meant to move and whether it did. It is only as good as the notes you give it, so keep a log.
Sources
- Noah Kagan on X: the audit of 67 shipped changes and the prompt
- n8n template: weekly narrative client reports with Google Sheets, Claude, and Gmail
- n8n template: forecast sales trends and weekly reports with Stripe, Sheets, Slack, and Gemini AI
Want a weekly "did it move a number" report built for your business? Book a free automation call at https://solvex.com.np.