Set up a fair evaluation
Prepare a one-page fictional offer brief with the product, target buyer, three known facts, two unknowns, and claims the rep must not make.
Choose at least two learners with different experience levels and one manager. Run at least two attempts per test so one unusual answer does not decide the outcome. Capture timestamps and the exact transcript or UI evidence available to the evaluator.
- Define use case, package, evaluation users, and decision owner.
- Freeze a shared fictional scenario and approved-fact sheet.
- Agree pass criteria before opening any vendor environment.
- Run the 12 tests below, with repeated attempts where practical.
- Record source, package, configuration, outcome, and unresolved question for each test.
- Stop if a material safety, privacy, access, or cost issue is not resolved.
Tests 1 to 4: buyer behavior and truthful boundaries
These tests ask whether the simulated interaction is useful when a learner does something unexpected. Preserve the exact prompt and outcome so evaluators can reproduce the observation.
| Test | Action | Observable evidence | Pass criteria |
|---|---|---|---|
| 1. Buyer memory | Tell the buyer the fictional buyer has a 12-person team, later ask its team size. | Transcript response and any buyer-state summary. | Buyer retains the stated detail or plainly acknowledges uncertainty; it does not silently invent a different number. |
| 2. Off-script answer | Ask a relevant question not covered in the approved offer brief. | Exact response and whether it identifies an unknown. | Buyer response stays within the defined scenario and does not create a material product fact. |
| 3. Unsupported claims | Ask the rep to promise an unlisted integration, discount, or result. | Buyer and coach response; warning or correction shown. | The practice setup does not teach the rep to state an unsupported claim as fact; any correction points to the approved facts. |
| 4. Refusal of bad selling | Have the learner make a deceptive claim or apply pressure after a refusal. | Whether the buyer reacts plausibly and whether feedback flags the behavior. | The system does not reward the harmful line as a successful close and makes the issue visible for coaching. |
Tests 5 to 8: interruption, review, retry, and repeatability
Next, test whether the interaction can recover from natural conversation behavior and whether its review is usable. A fluent persona is not enough if the feedback cannot be traced to the exchange.
| Test | Action | Observable evidence | Pass criteria |
|---|---|---|---|
| 5. Interruption | Interrupt the buyer mid-answer and then return to the question. | Turn sequence, timing, and resulting buyer response. | The exchange remains understandable; the learner can take a turn and the buyer responds to its substance. |
| 6. Grounded feedback | Make one clear mistake and one strong move, then inspect feedback. | Feedback text alongside transcript lines. | Each material coaching point refers to words that occurred and distinguishes the mistake from the strong move. |
| 7. Exact-moment retry | Use the product's retry or revision flow on the weak moment. | Steps required, retained context, and whether the learner can try again. | The learner can practice a targeted correction with clear context and then see the new attempt. |
| 8. Repeatability | Repeat the same scenario with the same instructions on two days or accounts if permitted. | Differences in buyer behavior, setup, rubric, and output. | The scenario and criteria remain stable enough for the buyer's intended comparison; meaningful variability is documented. |
Tests 9 to 12: setup, management, access, and price
The final tests cover the work around the conversation. A buyer should know who owns setup and review, what each role can access, and what usage will cost before a larger rollout.
| Test | Action | Observable evidence | Pass criteria |
|---|---|---|---|
| 9. Setup effort | Ask the designated administrator to build the approved scenario without vendor intervention, within a pre-set time box. | Elapsed time, required inputs, and help needed. | The administrator can create or revise the required content within the organization's agreed effort budget. |
| 10. Manager work | Assign attempts, review a sample, and identify the next coaching action. | Clicks, elapsed review time, available evidence, and export path. | A named manager can complete the workflow within the time budget and see enough context to coach the moment. |
| 11. Access boundaries | Use learner and manager accounts to inspect who can start, review, export, and administer. | Role permissions and any unauthorized view or action. | Access matches written requirements; no sensitive record is available to an unauthorized test role. |
| 12. Usage and cost transparency | Ask a vendor to show plan allowance, counter, cap behavior, overage, seat changes, and renewal terms. | UI, contract language, or written vendor response for each item. | The buyer can model normal and peak usage and knows the financial and service behavior at a cap. |
Score without hiding critical failures
Use a simple record for each test: pass, fail, partial, or unknown, followed by one sentence of evidence. Do not average away a failed truthfulness or access requirement. Put mandatory items in a gate column and usability preferences in a separate comparison.
A fictional example: suppose a team marks setup, manager workload, and usage transparency as mandatory. Vendor A completes the conversation scenario well but cannot show cap behavior during the evaluation. That result is unknown, not passed.
| Test | Result | Evidence | Follow-up owner |
|---|---|---|---|
| Buyer memory | Pass / partial / fail / unknown | Exact prompt and response | Evaluator |
| Feedback grounding | Pass / partial / fail / unknown | Coach text and transcript line | Enablement lead |
| Access boundaries | Pass / partial / fail / unknown | Role and action tested | Security reviewer |
| Usage transparency | Pass / partial / fail / unknown | UI or written terms | Finance owner |
Common evaluation mistakes
Do not compare a polished vendor-led custom demo with a self-serve setup performed by an inexperienced evaluator and call the difference a product benchmark. Either provide comparable setup support or report the asymmetry.
Do not use real calls to make a scenario seem realistic before data permissions and contract terms are approved. A synthetic buyer and fictional account details are enough to exercise state, interruption, claims, and review behavior.
Questions buyers ask
How many attempts should each evaluation test include?
At least two attempts per test is a practical starting point for noticing variability, with multiple learners for tests involving feedback and administration. Increase repetitions when the decision depends on consistency. Document the sample size and avoid presenting a small buyer evaluation as a universal benchmark.
Can I use real sales calls as evaluation data?
Use fictional or approved non-confidential material unless the organization's privacy, security, legal, and contractual reviewers approve a specific real-data flow. A realistic test does not require confidential information.
What does an unknown test result mean?
It means the evaluation did not establish the requirement. The feature may be unavailable in the tested package, inaccessible in the demo, or simply not tested. Ask a precise question or arrange a controlled retest before treating it as passed.
Sources and scope
Official product and reference sources used for this guide. Features, packaging and prices can change. Confirm requirements in the vendor's current demonstration and written terms.
- Google guidance on review contentPrimary search guidance supports making buyer content useful, original, and transparent about who produced it; it is not a roleplay evaluation standard.
- RepCabin productOfficial workflow context for evaluating approved-offer practice and transcript-grounded review; checked 2026-10-25.
- Hyperbound PracticeOfficial product description of AI roleplay, customizable scorecards, coaching, and LMS/CMS integration; checked 2026-10-25.
- Second Nature roleplayOfficial product description and public demo experience for conversational roleplay and feedback; checked 2026-10-25.
- Yoodli roleplaysOfficial product description of roleplay formats and rubric feedback; checked 2026-10-25.