# Enterprise AI pilot evaluation worksheet

Version: 10 November 2026

A reusable planning and review worksheet by Scott Lee Siang Fong.
Guide: https://leesiangfong.com/blog/enterprise-ai-roadmap/

Copy this file and replace the blank fields. This is a planning aid, not evidence of certification, legal compliance or guaranteed results. Set acceptance criteria before testing. Keep sensitive case details in an approved system and reference them here by ID.

## 1. Define the decision

- Workflow and people involved:
- Business problem:
- Intended benefit:
- Business owner:
- Technical owner:
- Source-content owner:
- Decision approver:
- Who can pause the pilot:
- What the AI may read:
- What it may draft:
- Actions it must not take:
- Human approval required before:
- Pilot dates:

## 2. Record the evidence and access boundaries

| Source ID | Document/version | Owner | Approval date | Allowed roles | Review date |
|---|---|---|---|---|---|
| | | | | | |

- Handling of missing or conflicting information:
- Escalation owner and route:
- Separation of private and public information:
- Retention and deletion rules:
- Relevant policies or requirements to verify:

## 3. Agree acceptance criteria before testing

| Measure | Definition and denominator | Required result | Failure that blocks launch | Evidence owner |
|---|---|---|---|---|
| Fully correct draft | Correct and complete against approved evidence, before human correction / all evaluated cases | | | |
| Appropriate escalation | Cases correctly referred because evidence or authority is insufficient / cases requiring escalation | | | |
| Unsupported policy claims | Count and severity; include promises, prices, eligibility and refunds where relevant | | | |
| Citation support | Claims supported by the cited source / claims checked | | | |
| Permission boundary | Unauthorised disclosures / named role-and-resource tests | | | |
| Staff effort | Total staff-minutes including review, correction and escalation / all cases | | | |
| Operating cost | Defined recurring costs / period or completed cases | | | |

A good average does not override a blocking failure. Write down which errors require stopping or revising the pilot even when other targets pass.

## 4. Build the comparison

- Test-case count:
- How cases represent ordinary, difficult and out-of-scope work:
- Baseline process and dates:
- AI-assisted process and dates:
- Matching method for case difficulty and staff experience:
- Task order and how repeated-case learning is handled:
- People doing the work:
- Reviewers and how disagreements are resolved:
- Approved answer or escalation for each case:
- Cases reserved for evaluation after revisions:
- Model, prompt, retrieval and application versions:

### Per-case record

| Case ID | Method | Source/version | Search + drafting staff-minutes | Review + correction staff-minutes | Escalation staff-minutes | Total staff-minutes | Correct before correction? | Escalation required and correct? | Unsupported claim or access issue | Reviewer |
|---|---|---|---|---|---|---|---|---|---|---|
| | Baseline / AI-assisted | | | | | | | | | |

Use mutually exclusive time categories so work is not counted twice. Count each person's effort; report elapsed customer turnaround separately if multiple people work at once. Define how waiting time is handled and apply the same rule to both methods.

### Permission tests

| Test ID | Role | Requested information/action | Expected boundary | Observed result | Pass/fail | Evidence reference |
|---|---|---|---|---|---|---|
| | | | | | | |

Passing a finite test set does not prove that all possible permissions or adversarial cases are safe.

## 5. Calculate and interpret results

| Measure | Baseline | AI-assisted | Difference | Meets agreed criterion? |
|---|---|---|---|---|
| Cases evaluated | | | | |
| Total staff-minutes | | | | |
| Staff-minutes per case | | | | |
| Review and correction minutes (part of total) | | | | |
| Fully correct drafts / all cases | | | | |
| Appropriate escalations / cases requiring escalation | | | | |
| Blocking failures | | | | |
| Recurring costs for comparable volume | | | | |

- Average staff-minutes per case = total staff-minutes / case count.
- For equal case counts: time reduction (%) = (baseline total - AI-assisted total) / baseline total × 100.
- For unequal case counts: compare average staff-minutes per case, provided case mix is comparable; do not compare totals as if volumes matched.
- Fully correct draft rate (%) = fully correct drafts / all evaluated cases × 100.
- Correct escalation rate (%) = correct escalations / cases requiring escalation × 100. Report not applicable if no cases require escalation.
- Review and correction time is already part of total staff effort. Do not deduct it a second time.
- If a denominator is zero, report the measure as unavailable rather than calculating a percentage.

### Costs and capacity

- Preparation, integration and training staff-hours:
- One-off external costs:
- Recurring model, hosting, licence and support costs:
- Ongoing content maintenance and evaluation effort:
- Unit-cost assumptions and source dates:
- Is freed staff capacity usable elsewhere? Explain:
- Does any reduction actually change cash expenditure? Explain:

Staff capacity released is not automatically cash saved. Separate one-off investment, ongoing costs and any demonstrated financial effect.

### Limits of the comparison

- Differences in case mix, staffing or test order:
- Missing baseline measures:
- Sample size and unusual cases not covered:
- Other changes during the pilot:
- What these observations support:
- What they do not establish, including causal claims:

## 6. Record the launch decision

Decision: [ ] Proceed with limits  [ ] Revise and retest  [ ] Stop

- Evidence supporting the decision:
- Blocking failures and required fixes:
- Unresolved risks or uncertainty:
- Approved users and workflow scope:
- Human review and excluded actions:
- Owner of each follow-up action:
- Approver and approval date:
- Next review date:
- New evaluation cases required after fixes:

## 7. Maintain the workflow

- Support owner:
- How corrections become approved source updates:
- Who checks source freshness:
- Changes that trigger retesting:
- Monitoring and incident escalation:
- Pause/rollback procedure:
- Outcome review: what actually happened after approval:

Keep the decision, its evidence and the later outcome together. Revise the conclusion when new evidence changes it.