Section III, cross-cutting discipline

Continuous improvement

Governance has to change as technology, threats and the business change. This discipline is the home of the lifecycle's loop: it makes sure that what is learned while agents run reaches the people who design and build the next version, and that governance itself keeps improving.

The loop from Change back to Design

Agents are not finished when they are deployed. Incidents, evaluation results, cost findings and feedback from partners all come out of Run time. Change time acts on them, and any change that alters what an agent does, reaches or may spend goes back through Design and Build.

1PlanObjectives confirmed again 2DesignScope reviewed again 3BuildNew version built and proven 4RunFindings recorded 5ChangeFindings acted on 6DecommissionThe loop ends

Every finding leads to a change, a new control, or a recorded decision to leave things as they are.

The rule for this discipline is that no finding is left without an answer. Each one is linked to the design or build change it caused, to a new control, or to a recorded decision not to change anything, with the reason for it.

Improving governance itself

Health and maturity

The organization watches measures that show when its controls are failing, and assesses the maturity of its governance each year, with an outside party at the higher tiers.

Lessons learned

Each incident and audit finding leads to a lesson, and each lesson leads to a change in a standard, a test or a control.

Keeping the framework current

The organization's framework is reviewed each year and after any serious incident, and every revision records what changed and why.

Governance as code

Controls are written as code wherever they can be, so they apply the same way every time and produce their own evidence. Policy written as code is versioned and reviewed like any other change.

Control objectives

There are 7 control objectives here. Each one states what it asks for, the risk it addresses and how it is measured. It then says what each risk tier requires, what evidence proves it, and who may check that evidence. The Operating Model page explains the tiers.

IMP-01

Governance health

Watch the health of governance itself, through measures that show when controls are failing.

Risk addressed: Governance failures nobody notices
Key metric: Share of governance measures reported on time

What each tier requires

Tier 1 Exploratory
The sandbox is reviewed each quarter.
Tier 2 Operational
Measures for each discipline are reported each quarter.
Tier 3 Critical
Measures are reported each month, with thresholds that trigger review.
Tier 4 Strategic
Measures are reported to the governance council each month.

Evidence: The governance measures and their history.
Who may check it: Internal audit.

IMP-02

Maturity assessment

Assess the maturity of governance regularly, to find gaps and improve.

Risk addressed: Governance that stops improving
Key metric: Assessments completed on schedule

What each tier requires

Tier 1 Exploratory
No assessment is required for the sandbox alone.
Tier 2 Operational
Maturity is assessed each year.
Tier 3 Critical
Maturity is assessed each year by someone outside the governance team.
Tier 4 Strategic
Maturity is assessed each year by an outside party.

Evidence: The assessment reports and the actions taken.
Who may check it: Internal audit and an outside auditor.

IMP-03

Comparison with others

Compare governance practice with published practice elsewhere, to find improvements.

Risk addressed: Falling behind good practice
Key metric: Gaps found and closed

What each tier requires

Tier 1 Exploratory
No comparison is required.
Tier 2 Operational
Practice is compared with published guidance each year.
Tier 3 Critical
Practice is compared with peers each year.
Tier 4 Strategic
The comparison is reported to the governance council with a plan.

Evidence: The comparison and the plan that followed.
Who may check it: Internal audit.

IMP-04

The framework kept current

Revise the organization's framework when technology, threats or the business change, and record what changed and why.

Risk addressed: Governance that does not adapt
Key metric: Time since the last framework revision

What each tier requires

Tier 1 Exploratory
The sandbox rules are revised when needed.
Tier 2 Operational
The framework is reviewed each year.
Tier 3 Critical
The framework is reviewed each year and after any serious incident.
Tier 4 Strategic
Framework revisions are approved by the governance council.

Evidence: The framework's version history with reasons.
Who may check it: Internal audit and an outside auditor.

IMP-05

Lessons learned

Capture lessons from incidents and audits, and feed them back into Design and Build, so that mistakes are not repeated.

Risk addressed: Repeated mistakes
Key metric: Share of incidents with a lesson captured and acted on

What each tier requires

Tier 1 Exploratory
The builder notes what went wrong.
Tier 2 Operational
Each incident has a written lesson.
Tier 3 Critical
Each lesson leads to a change in a standard, a test or a control, or a recorded reason for none.
Tier 4 Strategic
Lessons are shared across business units and reviewed by the governance council.

Evidence: The lessons log and the changes it led to.
Who may check it: Internal audit.

IMP-06

Governance as code

Turn controls into code where they can be, so that they are applied the same way every time and produce their own evidence.

Risk addressed: Manual governance that slows work and varies
Key metric: Share of controls enforced and evidenced by code

What each tier requires

Tier 1 Exploratory
Sandbox limits are enforced by the platform.
Tier 2 Operational
Release gates are automated.
Tier 3 Critical
Run-time limits on identity, reach, spend and stopping are enforced by code outside the agent.
Tier 4 Strategic
Policies are written as code, versioned and reviewed like any other change.

Evidence: The list of controls with how each is enforced, and the policy code's history.
Who may check it: Internal audit and an outside auditor.

IMP-07

The loop from Change back to Design

Make sure what is learned at Run and Change reaches Design and Build: incidents, evaluation results, cost findings and partner feedback each lead to a change, a new control or a recorded decision to leave things as they are.

Risk addressed: The same problems built into the next version
Key metric: Share of run-time findings that reached a design or build decision

What each tier requires

Tier 1 Exploratory
The builder carries findings into the next experiment.
Tier 2 Operational
Findings are recorded against the agent and reviewed at the next release.
Tier 3 Critical
Each finding is linked to the design or build change it caused, or to a decision not to change.
Tier 4 Strategic
The loop is reviewed each quarter by the governance council across all Tier 4 agents.

Evidence: Findings linked to the changes or decisions that followed them.
Who may check it: Internal audit and an outside auditor.

Quick reference

ID	Objective	Key metric
IMP-01	Governance health	Share of governance measures reported on time
IMP-02	Maturity assessment	Assessments completed on schedule
IMP-03	Comparison with others	Gaps found and closed
IMP-04	The framework kept current	Time since the last framework revision
IMP-05	Lessons learned	Share of incidents with a lesson captured and acted on
IMP-06	Governance as code	Share of controls enforced and evidenced by code
IMP-07	The loop from Change back to Design	Share of run-time findings that reached a design or build decision