Most AI pilots look good in a demo and then stall. A team connects a model to its CRM or inbox, tries ten examples, likes what it sees, and rolls it out. Three weeks later, a few wrong answers reach customers and trust collapses. The missing piece is almost always the same: no AI workflow evaluation before scaling. This playbook shows how a French B2B company can test an AI workflow the way it would test any business process, with clear criteria, a realistic test set and a simple go/no-go rule.
Why AI workflows fail after a good demo
A demo tests the happy path. Production brings incomplete emails, angry customers, duplicate records, scanned PDFs and questions nobody planned for. Language models are also non-deterministic: the same input can produce slightly different outputs, and a prompt tweak or model update can quietly change behaviour.
That is why the major model vendors put evaluation at the centre of their guidance. Anthropic's documentation on defining success criteria and building evaluations asks teams to set specific, measurable goals before tuning prompts. OpenAI's evaluation best practices describe evaluation as a continuous process rather than a one-off check. Both apply whichever model you use, including open-weight models you host yourself (see our self-hosted LLM vs frontier API guide).
Typical warning signs that a workflow was never properly evaluated:
- Nobody can say what "good" means: the success criterion is "it looks right".
- The test set is ten hand-picked examples: no edge cases, no bad inputs.
- Changes are shipped blind: a new prompt or model goes live without re-running the same tests.
- Errors are found by customers: there is no review step and no log of failures.
Step 1: Define success criteria the business can sign off
Start with the business owner, not the prompt. For each workflow, write three to five criteria that are specific and measurable. Anthropic recommends criteria that are specific, measurable, achievable and relevant, and notes that most use cases need several dimensions at once, such as accuracy, consistency, tone, privacy, latency and cost.
Example for an AI workflow that drafts replies to inbound customer requests and logs them in the CRM:
- Accuracy: the reply answers the actual question and does not invent prices, dates or commitments.
- Routing: the request is tagged with the right category and owner in the CRM.
- Tone and language: French replies use the formal "vous" and the company's wording.
- Privacy: no personal data from another customer appears in the draft.
- Operations: the draft is ready in under a minute and the cost per request fits the budget.
Then set a threshold for each criterion before you run any test, for example "routing correct in at least 95% of cases, zero invented commitments". Agreeing on thresholds in advance stops the team from moving the goalposts once results come in.
Step 2: Build a test set from real work
A useful test set mirrors what the workflow will really see. Pull 50 to 200 real, anonymised examples from your history (emails, tickets, CRM notes) and add deliberate edge cases. Both vendors' guides insist on covering typical, edge and adversarial cases, and Anthropic adds that many automatically graded cases usually beat a handful of hand-graded ones.
A practical checklist for your first test set:
- Typical cases: the 70% of requests the team handles every day.
- Missing or messy input: empty fields, forwarded threads, attachments only.
- Very long input: long email chains or multi-page documents.
- Ambiguous requests: two questions in one message, unclear customer identity.
- Out-of-scope requests: topics the workflow must refuse or hand to a human.
- Sensitive cases: complaints, legal threats, personal data requests.
For each example, store the expected result: the correct category, the key facts the answer must contain, or simply "escalate to a human". This reference set becomes a company asset, and it is reusable when you switch models or vendors.
If the workflow uses tools, for example an assistant connected to your CRM through MCP as in our Claude and CRM meeting-prep walkthrough, also test the actions: did it read the right record, call the right tool, and avoid writing where it should only read? OpenAI's guide to evaluating agent workflows recommends starting from traces, the end-to-end record of model calls, tool calls and handoffs, and grading them to spot wrong tool selection or policy violations.
Step 3: Choose how to grade, then decide go or no-go
There are three ways to score outputs, and most companies combine them:
- Rule-based checks: exact or keyword matches, valid JSON, correct CRM category. Fastest and most reliable, but limited to what can be checked mechanically.
- Human review: a business expert grades outputs against the criteria. Highest quality, but slow and costly, so use it to build the reference answers and to audit samples.
- Model-graded checks ("LLM as a judge"): a second model scores outputs against a written rubric. Fast and flexible for tone or completeness, but it must be validated against human grades first.
Use the model-graded approach carefully. OpenAI notes that judge models can favour longer answers or a particular position, and recommends pass/fail or pairwise grading, asking for reasoning before the score, and checking agreement with human labels. Anthropic suggests grading with a different model from the one that produced the output.
A simple go/no-go rule for a pilot:
- Go to production if every criterion meets its threshold, no critical failure (invented commitment, data leak) appears, and human reviewers agree with the automated grades on a sample.
- Go with a human in the loop if quality is close to target but critical cases still need checking: the AI drafts, a person approves.
- Stop or redesign if critical failures persist, or the workflow only passes on the typical cases and fails on edge cases.
Step 4: Keep evaluating after go-live and train the team
Evaluation does not end at launch. Re-run the same test set on every prompt change, model upgrade or new data source, and keep adding real failure cases from production logs, as OpenAI recommends. A monthly review of a small sample of live outputs by the business owner is often enough to catch drift early.
People matter as much as metrics. Under Article 4 of the EU AI Act, providers and deployers of AI systems must take measures to ensure a sufficient level of AI literacy among the staff who use them. According to the European Commission's AI literacy Q&A, this obligation has applied since 2 February 2025, national authorities supervise it from 2 August 2026, and no certificate is required; keeping internal records of training is a sensible practice. Teaching reviewers how the workflow is evaluated, and what to do when it fails, covers part of that obligation in a very concrete way.
What to do now
A four-week plan most mid-size companies can run:
- Week 1: pick one live or planned AI workflow, name a business owner, and write its success criteria and thresholds.
- Week 2: collect 50 to 200 anonymised real examples plus edge cases, with expected results.
- Week 3: run the workflow on the test set, grade with rules plus a human sample, and add a validated model grader if volume requires it.
- Week 4: apply the go/no-go rule, document the results, train the reviewers, and schedule re-runs for every future change.
Evaluating an AI workflow is not a research exercise. It is the same discipline as testing any business process: agree on what good looks like, test on real cases, measure, then decide. Companies that build this habit can change models, prompts or vendors with confidence, because they have a reference test set that tells them, in hours rather than weeks, whether the new version is better or worse.
If you want an outside view, ASC Digital can review existing pilots through an AI audit, design and test the workflow with you as part of our AI workflow implementations, and train the people who will run it through AI enablement and training.
