AI assurance for defence should evaluate the complete socio-technical system: models, data, software, infrastructure, people, procedures, interfaces, suppliers and operating conditions.
Model evaluation is necessary, but it is not sufficient for high-consequence AI. A model can perform well in a benchmark and still become unreliable when integrated into a larger system with changing data, complex permissions, automation, communications constraints and human operators. Defence assurance therefore needs to move from the question “does the model work?” to the question “what evidence supports trusting this system for this defined use, under these conditions, at this point in time?”
01 Executive summary
Assurance is the disciplined process of developing and maintaining justified confidence in a system. For AI-enabled defence and critical systems, that confidence cannot be reduced to a single accuracy score, certification label or red-team exercise. It must connect intended use, technical architecture, evaluation, cybersecurity, human factors, configuration control, runtime evidence and accountable acceptance.
The most important shift is from model-centric evaluation to system-level assurance. The object being trusted is not the model in isolation; it is the integrated capability and the organization operating it.
02 Assurance is a lifecycle, not a deployment gate
Traditional acceptance processes often concentrate evidence near deployment. AI-enabled systems challenge that model because behaviour can change when data changes, models are updated, prompts and tools evolve, external services change, or an agent receives new permissions. Assurance must therefore continue through operation.
A mature lifecycle connects four activities: defining the intended use and boundaries; testing against those boundaries; preserving traceability about what changed; and monitoring the operating system for signals that invalidate prior assumptions. The result is not continuous re-certification of everything. It is a risk-based mechanism for deciding when existing evidence remains valid and when deeper review is required.
03 Define the assurance claim before collecting evidence
An assurance case is a structured argument supported by evidence. The top-level claim should be narrow enough to evaluate. Instead of asserting that a model is “safe” or “trusted,” an assurance case might state that a defined system is acceptable for a specified decision-support function, with named users, data sources, operating conditions and required human review.
That framing forces clarity about scope. Which model version is included? Which external tools can be invoked? What authority does the system have? Which environments are in scope? What assumptions are made about connectivity, sensors or users? What failure conditions are unacceptable? Without this definition, a collection of tests can look impressive while failing to support a meaningful decision.
04 Five evidence layers
System definition
Assurance begins with an inventory of the actual system: models, versions, interfaces, data sources, tools, identities, permissions, infrastructure, human roles and external dependencies. Without a stable definition of the assessed system, test results can quickly become disconnected from what is running.
Evaluation evidence
Evaluation should be tied to intended use and credible failure modes. Evidence may include task performance, robustness under variation, boundary conditions, misuse testing, adversarial testing, data-quality analysis, scenario testing and comparisons to non-AI alternatives. The evaluation should record configuration and data conditions so results can be interpreted later.
Security evidence
AI systems create new paths for data exposure, unauthorized tool use, prompt or instruction manipulation, model extraction and supply-chain risk. Assurance therefore needs security evidence covering identity, authorization, secrets, software dependencies, interfaces, logging, incident response and the ways an AI component can influence other systems.
Human and procedural evidence
Human oversight is only meaningful if operators have enough information, time, authority and training to intervene. Evidence can include interface testing, procedure exercises, escalation paths, decision records and observation of how people actually use the system under realistic workload and time pressure.
Operational evidence
Once deployed, telemetry, incidents, model changes, exceptions, performance drift, policy violations and operator feedback become part of the assurance record. Operational evidence is what allows the organization to determine whether the original acceptance rationale still holds.
05 Testing, evaluation, verification and validation
NATO’s revised AI strategy identifies the development of an Alliance-wide testing, evaluation, verification and validation landscape as a desired outcome for responsible AI adoption. NIST’s AI Resource Center similarly emphasizes testing, evaluation, verification and validation as part of operationalizing AI risk management. The terminology differs across institutions, but the underlying idea is consistent: confidence should be built through multiple forms of evidence rather than a single benchmark.
For an integrated system, testing may examine observable behaviour; evaluation may assess fitness for a defined use; verification may ask whether implementation conforms to requirements; and validation may ask whether the system is suitable in the intended operational context. The boundaries between these activities vary, but keeping the questions distinct helps prevent one strong result from being overinterpreted.
06 Security and supply-chain assurance
AI assurance and cybersecurity cannot be separated cleanly. The behaviour of an AI-enabled system can be altered by compromised software, manipulated data, stolen credentials, untrusted plugins, external APIs or unauthorized configuration changes. A credible assurance case therefore needs to identify critical suppliers, model and software provenance, privileged access, update mechanisms, network trust boundaries and incident-response dependencies.
Supply-chain evidence should also account for change. A vendor service may update without the customer’s deployment process changing. An open-source component may introduce a vulnerable dependency. A model provider may release a new version with different behaviour. Assurance should define which external changes are material enough to trigger review.
07 Human factors are part of system assurance
Human oversight is often stated as a principle but under-specified as a control. The relevant questions are operational: What does the operator see? Can the operator understand system confidence and constraints? How much time is available to intervene? Does the interface surface degraded data or communications? Can the operator override, pause or revert the system? Is intervention logged?
These are engineering questions as much as governance questions. If the technical interface does not provide usable information or a reliable control path, a policy stating that a human is responsible may not produce meaningful oversight.
08 Runtime assurance and monitoring
Continuous assurance does not mean continuously re-certifying an entire system. It means designing the architecture so changes, evidence and operational signals can be observed and routed to proportionate review. Low-risk configuration changes may require lightweight checks; a major model, data, permission or integration change may trigger deeper evaluation.
Runtime monitoring can track model or software versions, authorization changes, tool calls, performance signals, exception rates, data-quality indicators, operator overrides and security events. The useful question is not “how much can we log?” but “which signals would tell us that an important assurance assumption has changed?”
09 Configuration and change control
Assurance has a time dimension. A result is only meaningful for the configuration that was tested. Organizations therefore need a configuration baseline connecting model version, prompt or policy configuration, tool permissions, data sources, software dependencies and infrastructure. Material changes should be traceable to an owner, reason, evaluation decision and release record.
This is especially important for AI services delivered through external APIs, where model behaviour or provider infrastructure can change independently of the local application. Supplier notification, monitoring and contingency provisions become part of the assurance strategy.
10 Independent challenge and acceptance authority
Technical teams should not be the only function deciding whether their own system is ready for consequential use. Mature assurance introduces independent challenge proportionate to risk: security review, model-risk review, safety analysis, legal review, operational evaluation or governance challenge. Independence does not always require an external auditor, but it does require enough separation for assumptions and evidence to be questioned credibly.
The final acceptance decision should also be explicit. Who has authority to accept residual risk? What evidence must be available? What limitations are recorded? When does approval expire or require reconsideration? These governance mechanics convert assurance from a technical report into an operating discipline.
11 Assurance requirements in procurement
Buyers can strengthen assurance by asking for evidence before integration. Procurement requirements can address model and software versioning, change notification, security testing, data handling, logging, portability, incident support, subcontractors, evaluation artifacts and access to technical documentation. The objective is not to force every supplier into one framework; it is to ensure that the buyer can build a defensible system-level assurance case.
Where evidence cannot be provided, the absence itself becomes a risk input. Leadership can then decide whether to compensate through architecture, restrict the use case, require additional testing or select a different approach.
12 Research view
Canadian AI Defence treats assurance as an evidence architecture connecting design intent, technical controls, evaluation, security, human authority and operational state. The objective is not to prove that an AI-enabled system is universally safe. It is to create a scoped, dated and reviewable basis for deciding whether a defined system is acceptable for a defined use.
This approach is compatible with emerging public frameworks but is not a substitute for formal certification, legal review, safety engineering, security accreditation or government approval where those processes apply.
13 Selected public references
- NIST AI Risk Management Framework
- NIST AI Resource Center
- DND/CAF Artificial Intelligence Strategy
- Summary of NATO’s Revised Artificial Intelligence Strategy
This publication discusses high-level assurance architecture and governance. It does not provide certification, legal or operational military advice and does not represent an assessment of any government or military system.


