Prompt sampling: why we test ten prompts and not one
Our free scan asks ten prompts, not one, and not a hundred. The number is a design decision with reasoning behind it, and since the rest of the method is public, this piece of it should be too.
Sampling, in owner terms
One question to one AI on one day is an anecdote. Ten designed questions, spread across the ways customers actually ask, are a sample: small, honest about its size, and capable of showing a pattern. The difference isn’t the effort. It’s that a sample is built to be representative and an anecdote just happened.
No statistics lecture needed beyond that. You already use sampling instinct when you taste the soup twice before deciding it needs salt.
How the ten get chosen
Intent spread first: emergency asks, research asks, cost asks, trust asks, because a single phrasing tells you almost nothing and each intent pulls different sources. Customer phrasing second: the questions are written the way people type them, not the way marketers file them. Industry variation third: a dental practice’s ten and a roofer’s ten share structure, not wording, because their customers don’t share questions either.
What ten can support, and what it can’t
Ten prompts can show a pattern: named often, named rarely, named on cost questions but not emergency ones, absent everywhere. That’s actionable, and it’s honest at that resolution.
Ten prompts cannot support precision claims, and we don’t make them. A move from 3 mentions to 4 out of ten is not “visibility up 33%”; at this sample size it’s one answer’s difference and gets treated as noise until it repeats. This is also why share of voice needs its prompt list published to mean anything: the sample defines the number, always.
Why not a hundred
Because the tradeoff stops paying. A hundred prompts per scan would cost ten times as much to collect and read, mostly duplicate the intents ten already cover, and tempt everyone involved toward precision theater: decimal-point movements on a number that’s still a sample. There’s also a quieter risk our industry doesn’t talk about: giant prompt sets make it easier to game the result, because somewhere in a hundred questions is a subset any business wins. Our opinion, defended: ten designed prompts reported honestly beat a hundred reported impressively, and the vendors running huge sets rarely publish which questions produced their numbers. Continuous tracking is the honest way to grow the sample: not more questions on one day, but the same designed questions, checked daily, until the pattern has a spine.