There are 12 control objectives here. Each one states what it asks for, the risk it addresses and how it is measured. It then says what each risk tier requires, what evidence proves it, and who may check that evidence. The Operating Model page explains the tiers.
PRF-01Accuracy benchmarks
Measure each agent's accuracy against known correct answers, so that a decline is noticed.
- Risk addressed
- Accuracy declining without anyone noticing
- Key metric
- Share of agents with an accuracy benchmark
What each tier requires
- Tier 1 Exploratory
The builder checks results by hand.
- Tier 2 Operational
Each agent has a benchmark set that runs at release.
- Tier 3 Critical
The benchmark runs on a schedule against live work, and a decline alerts the owner.
- Tier 4 Strategic
The benchmark set is reviewed by the value owner and refreshed each quarter.
- Evidence
- The benchmark set and its results over time.
- Who may check it
- Internal audit and an outside auditor.
Measure how quickly agents answer against the times promised to the people and agents that use them.
- Risk addressed
- Slow agents that frustrate their users
- Key metric
- Share of agents meeting their promised response time
What each tier requires
- Tier 1 Exploratory
No requirement applies to an experiment.
- Tier 2 Operational
Each agent states a response time and measures it.
- Tier 3 Critical
Missed response times alert the owner, and the cause is recorded.
- Tier 4 Strategic
Response times are reported to dependents and partners under their contracts.
- Evidence
- Response time measurements against the promised times.
- Who may check it
- Internal audit and a partner.
Compare how much work agents can handle with the demand on them, to find bottlenecks before users do.
- Risk addressed
- Bottlenecks in capacity
- Key metric
- Throughput compared with demand
What each tier requires
- Tier 1 Exploratory
No requirement applies to an experiment.
- Tier 2 Operational
Throughput and demand are measured.
- Tier 3 Critical
A forecast of demand is compared with capacity each quarter.
- Tier 4 Strategic
Capacity for peak demand is tested before known peak events.
- Evidence
- Throughput and demand measurements and the capacity forecast.
- Who may check it
- Internal audit.
Measure how much an agent's answers to the same question vary, so that its behavior is predictable.
- Risk addressed
- Unpredictable behavior
- Key metric
- Variation in answers to repeated questions
What each tier requires
- Tier 1 Exploratory
No requirement applies to an experiment.
- Tier 2 Operational
A sample of repeated questions is checked at release.
- Tier 3 Critical
Variation is measured on a schedule and kept within a stated range.
- Tier 4 Strategic
Variation outside the range blocks a release.
- Evidence
- The repeated-question results and the stated range.
- Who may check it
- Internal audit.
PRF-05Continuous evaluation
Evaluate agents continuously, so that a change in how good their answers are between releases is caught.
- Risk addressed
- Quality drifting between releases
- Key metric
- Evaluation coverage across running agents
What each tier requires
- Tier 1 Exploratory
The builder reviews sample outputs.
- Tier 2 Operational
An evaluation suite runs at every release.
- Tier 3 Critical
Evaluation also runs on a sample of live work, by an evaluator that is not the agent itself.
- Tier 4 Strategic
Evaluation results are part of each review and are shared with the value owner each month.
- Evidence
- Evaluation results for releases and for sampled live work.
- Who may check it
- Internal audit and an outside auditor.
PRF-06Regression detection
Detect a fall in performance caused by a model update or a configuration change, and alert the owner.
- Risk addressed
- Quality loss after a model update
- Key metric
- Time to detect a regression
What each tier requires
- Tier 1 Exploratory
No requirement applies to an experiment.
- Tier 2 Operational
The evaluation suite runs after any model or configuration change.
- Tier 3 Critical
A regression blocks promotion, and the previous version stays available.
- Tier 4 Strategic
Regressions are reported to the governance council when they reach production.
- Evidence
- Evaluation comparisons before and after each change.
- Who may check it
- Internal audit.
Measure how often agents state things that are not true, and keep the rate within a stated level.
- Risk addressed
- Factual errors that damage trust
- Key metric
- Rate of factual errors in sampled answers
What each tier requires
- Tier 1 Exploratory
The builder notes errors seen in testing.
- Tier 2 Operational
A sample of answers is checked for factual errors at release.
- Tier 3 Critical
Factual errors are measured on live work, and the rate is kept within a stated level.
- Tier 4 Strategic
Answers that make claims are checked against a source before they are sent.
- Evidence
- Sampled answers with error findings, and the measured rate over time.
- Who may check it
- Internal audit and an outside auditor.
Connect each agent's technical measures to the business objective it serves, so that technical success is not mistaken for value.
- Risk addressed
- Technical success without business value
- Key metric
- Share of agents whose objective measure is reported
What each tier requires
- Tier 1 Exploratory
The experiment states what it set out to learn.
- Tier 2 Operational
The objective's measure from section I is reported each quarter.
- Tier 3 Critical
The value owner compares the measure with cost each quarter.
- Tier 4 Strategic
The measure is reported to the executive sponsor with the agent's cost.
- Evidence
- The objective measure, its trend and the value owner's review.
- Who may check it
- Internal audit.
PRF-09Finding underperformers
Find agents that do worse than their benchmarks or than other agents doing similar work.
- Risk addressed
- Poor agents harming the business
- Key metric
- Time from decline to flag
What each tier requires
- Tier 1 Exploratory
No requirement applies to an experiment.
- Tier 2 Operational
Agents below their benchmark are flagged at review.
- Tier 3 Critical
Agents below their benchmark are flagged automatically and given a remedy plan.
- Tier 4 Strategic
Persistent underperformers are reviewed by the governance council for retirement.
- Evidence
- The list of flagged agents and their remedy plans.
- Who may check it
- Internal audit.
Track how long it takes to fix performance problems once they are found.
- Risk addressed
- Poor performance going on too long
- Key metric
- Mean time to remedy
What each tier requires
- Tier 1 Exploratory
No requirement applies to an experiment.
- Tier 2 Operational
Performance problems are tracked to closure.
- Tier 3 Critical
Problems have target remedy times by severity.
- Tier 4 Strategic
Missed targets are reported to the executive sponsor.
- Evidence
- The problem log with dates found and fixed.
- Who may check it
- Internal audit.
PRF-11Scheduled quality audits
Complete the scheduled performance and quality audits. The Assurance discipline sets the audit plan.
- Risk addressed
- Gaps from missed audits
- Key metric
- Share of scheduled audits completed
What each tier requires
- Tier 1 Exploratory
No audit is required for an experiment.
- Tier 2 Operational
Quality is audited at the agent's review date.
- Tier 3 Critical
Quality is audited each half year by someone outside the team.
- Tier 4 Strategic
Quality is audited each quarter, and an outside auditor reviews the results each year.
- Evidence
- The audit schedule and completed audit reports.
- Who may check it
- Internal audit and an outside auditor.
Collect feedback from the people and agents that use an agent, and use it to improve the agent.
- Risk addressed
- Signals for improvement that nobody collects
- Key metric
- Share of feedback acted on
What each tier requires
- Tier 1 Exploratory
The builder asks users directly.
- Tier 2 Operational
Users can give feedback, and the owner reviews it.
- Tier 3 Critical
Feedback is tracked to an action or a reason for no action.
- Tier 4 Strategic
Feedback trends are reported with the agent's evaluation results.
- Evidence
- The feedback log and the actions taken.
- Who may check it
- Internal audit.