Let the AI score your skill against known-good inputs, propose a better version, and iterate until the score holds. The skill improves itself.
An eval — short for evaluation — runs your skill against a fixed set of inputs and scores each output against measurable criteria. It turns "is this skill any good?" into a number. With a number, you can see whether a change helped, hurt, or did nothing.
In the demo, the author uses a small library of three real YouTube transcripts as inputs. The skill under test — a YouTube description generator — runs against each transcript. Each output is graded on hook quality, timestamps, tone, structure, and hallucinations. Five tests per criterion, scored pass/fail.
The auto-research skill (built by Marcus on the SharePoint AI team) is a skill that improves other skills. You point it at a target skill, give it inputs to test against, and define what "good" looks like. It runs the baseline, proposes a mutation, reruns, and keeps iterating until the score plateaus or hits your target.
You invoke it with one big prompt that names the target skill, the input set, the criteria, and an output convention (e.g. "save the improved version as V2"). Then you watch it work. The model is both the engineer who edits the skill and the judge who scores the result.
Each criterion is a yes/no question with a defined pass/fail bar. For the description skill the author used: "Do the first one to two sentences create curiosity or clearly state value?" Pass = yes. Fail = no. Specific enough that even the model judging can answer it the same way twice.
The author cheats — productively — by asking Copilot to draft an initial criteria list from a sample of inputs. The model knows what these transcripts look like, so it can suggest dimensions worth grading. You edit and approve, then hand it to the eval skill.
The loop has three steps. Baseline: run the skill on every input, score each output, record the total. Mutate: based on the failing scores, the skill edits its own prompt — rewording instructions, tightening anti-hallucination guardrails, restructuring sections. Rerun: same inputs, same criteria, fresh score.
In the demo, the baseline scored 85%. The first mutation lifted it to 100% on the first four inputs — but a follow-up rerun on a larger set exposed a remaining hallucination issue. The skill mutated again, reran, and finally held the improvement across the full set. Final lift: 15 percentage points.
Always tell the auto-research skill to save its improved output as a new version (V2, V3 …) rather than overwriting the original. The original is your rollback. If V2 turns out to regress on a real-world input the eval set didn't catch, you can swap back in seconds.
Naming convention: youtube-description.md stays. The optimizer writes youtube-description-v2.md. You compare side by side, promote V2 when you're confident, archive the original.