# AI pilot scorecard

By John Rice · Make AI useful.

Use this scorecard to decide whether one AI workflow earns a place in your business. Agree on the pass conditions before running the pilot. Record the complete task, including review and correction.

## 1. Define the decision

- Workflow and version:
- Business owner:
- Reviewer:
- Pilot dates:
- Intended output:
- Decision this pilot should support:
- Sample size and selection method:
- Why these examples represent the work:
- Limits of the sample:

## 2. Set acceptance conditions

Separate conditions every accepted output must meet from improvement targets. A faster draft still fails if it creates an unsupported commitment.

| Condition | Verification method | Required threshold |
| --- | --- | --- |
| Important facts are supported by the supplied sources | | |
| Required items are present | | |
| Missing or conflicting information is visible | | |
| Owners, dates, and amounts are supported | | |
| Reviewer can use the output for the next step | | |
| Access and actions stay within the agreed scope | | |
| Total recurring effort is acceptable | | |

- Critical failure that stops the pilot:
- Maximum acceptable review effort:
- Maximum acceptable correction effort:
- Person authorized to accept changes to these conditions:

## 3. Record the current process

Time a few representative examples from input collection to accepted output. Record active effort and elapsed turnaround separately. Include checking and rework.

| Example ID | Input condition | Active effort (min) | Elapsed turnaround (min) | Errors or omissions | Accepted? |
| --- | --- | --- | --- | --- | --- |
| | | | | | |
| | | | | | |
| | | | | | |

## 4. Record each pilot run

Use the same units across runs. For AI generation, record active supervision in the effort column; include unattended waiting in elapsed turnaround. Do not count the same minutes twice.

| Example ID | Workflow version | Preparation (min) | Active generation/supervision (min) | Review (min) | Correction (min) | Total active effort (min) | Elapsed turnaround (min) | Accepted? |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| | | | | | | | | |
| | | | | | | | | |
| | | | | | | | | |

Total active effort = preparation + active generation/supervision + review + correction.

Log the result separately so an attractive average cannot hide important failures.

| Example ID | Missing input or integration issue | Unsupported or incorrect claims | Required items omitted | Correction made | Critical failure? |
| --- | --- | --- | --- | --- | --- |
| | | | | | |
| | | | | | |
| | | | | | |

## 5. Account for cost and maintenance

- One-time setup effort:
- One-time setup cost:
- Tool, model, and infrastructure cost during the pilot:
- Cost per accepted output, if measurable:
- Recurring maintenance tasks and owner:
- Expected maintenance effort per period:
- Permission or integration issues still unresolved:

Do not report saved effort as realized revenue. If you estimate capacity, label it as an estimate and state the assumptions.

## 6. Review the evidence

- Examples attempted:
- Examples accepted without correction:
- Examples accepted after correction:
- Examples rejected:
- Critical failures:
- Typical effort and range for the current process:
- Typical effort and range for the pilot:
- What happened on incomplete and ambiguous examples:
- Whether the reviewer would choose to use this workflow again, and why:

Small samples provide a learning signal. Describe the sample and conditions when sharing results; do not present them as universal performance claims.

## 7. Decide and assign the next action

- Decision: continue / revise / expand / stop
- Reason, tied to the acceptance conditions:
- Specific weak step to change:
- Version or configuration change:
- Examples to rerun after the change:
- Next review date:
- Next owner and action:
- Conditions for expanding the pilot:

Keep the brief, inputs you are allowed to retain, workflow version, outputs, corrections, and decision together. This turns the pilot into evidence you can use later.