How to stop AI from making slop

Most marketers have never heard of evals. But evals are the answer to preventing AI generated slop.

An eval (short for evaluation) is just a grade on the work of AI. That's all.

Software engineers have widely adopted evals. Mostly out of necessity. There's just no other way to monitor AI output at scale.

Evals aren't even that complicated. In fact once you see how simple they are, you'll wonder how you lived without them for so long.

Grade your AI's work

Right now, marketers spend all their time on the inputs that go into AI. A lengthy prompt, a skill file, or if you're more advanced, a repository full of context.

Those are all great. But AI can be unpredictable. What you want is something concrete. Something that can make you feel like you have real control over the quality of your AI outputs. What you want are evals.

Start with PASS/FAIL

You can approach evals in two ways. First is a to grade the output on a scale (e.g 1 to 10), the other is just a pass or fail.

I prefer pass/fail to start. Grading on a scale sounds cool but you can get lost in the sauce pretty quickly. The whole point of evals is to introduce predictability in your system, so the simpler the grading the better.

LLMs are very well versed in evals because they are trained on them. So they can help you write great ones. Just talk to your AI about it.

The aim is to codify what determines a PASS or a FAIL.

Compare human judge to LLM judge

Let's say you built a rubric for eval-ing hooks. You then generate a bunch of hooks and have AI act as a judge based on your rubric.

Then, you separately grade them yourself too.

You'll notice that you and your AI judge didn't grade the exact same way.

The % of time you both agreed is your alignment score.

Ask AI to improve the rubric

Feed the results of the grading exercise back into your AI. Ideally with some commentary on the ones where you and AI graded differently. Explain why you graded the way you did.

Then ask your AI to update the rubric. You're letting AI self improve to meet your standard of judgment.

Keep going in batches. Repeat until you get to a high alignment score. Now you have real confidence in your AI system.

Then, flip your eval into a gate

Here's my favorite part. Once you have a rubric with a high alignment score, you can use it as a quality gate in your system.

Tell your AI: keep generating hooks, but discard all the ones that fail the eval.

Now the eval stops being just a report card. Your AI can go ham generating outputs but it throws away anything that doesn't pass your eval.

This allows you to give your AI more free reign on how it tackles the work. As long as it clears your evals, you're happy.

As you're relationship with AI matures, you start to spend less time micromanaging how it gets work done and more time aligning on what good looks like.

Reducing the friction to doing evals

Our agent @runneth has a lot of autonomy. It has its own computer and autonomously runs its own X account.

I have a channel set up with it in slack called #runneth-evals. Every time it engages with someone, it posts the interaction in that channel. I grade it with a PASS or FAIL and some commentary.

Runneth compares my grades to its own and constantly self-improves.

Evals for anything

You can use evals for anything. Grading hooks, QA-ing video ads, writing briefs, etc.

Next time you want to get AI to do anything, start with building the eval rubric for the work. Then watch you're entire approach to AI change!

Links

• x.com/rezakhadjavi/status/2064716716588868086