Begin with the mission, not the model

Model accuracy cannot establish operational fitness by itself. TEVV should begin with intended mission, operational design domain, consequence profile, human responsibilities, dependencies and conditions under which autonomy must be restricted or withdrawn.

BIT's developing approach is intended to translate these factors into testable requirements and explicit pass, fail and review criteria.

Test normal, boundary and failure conditions

A scenario catalogue can include sensor dropout or bias, conflicting data, communications loss, timing faults, invalid or revoked authority, human handover, component degradation, model updates and foreseeable misuse.

BUILD-UP

Analysis to HIL

Use analysis, simulation and hardware-in-the-loop before higher-consequence live evaluation.

CLAIM-BASED

Method follows risk

Select the test environment and evidence depth according to the claim and consequence.

Evaluate authority and degraded behaviour

TEVV should test not only what a system predicts, but what it is permitted to do. Tests should cover correct ALLOW, bounded CONSTRAIN or DEGRADE, appropriate HOLD, rejection through DENY and defined ABORT.

Preserve configuration and test provenance

A result is difficult to reproduce without knowing the hardware, software, model, data, policy, authority, scenario, environment and instrumentation used.

BIT is developing evidence structures intended to bind these elements to each execution and observed result.

Continue TEVV after initial acceptance

Behaviour can change when models, sensors, infrastructure, data distributions, suppliers or policies change. BIT's developing approach includes change-triggered reassessment and regression testing so earlier evidence is not assumed to remain valid after a material change.

Frequently asked questions

What does TEVV mean?

Test and evaluation, verification and validation examine whether requirements are met, the system was built as specified and the capability is fit for intended use.

Why is model testing insufficient?

Operational behaviour emerges from the complete system: data, sensors, software, hardware, communications, people, policy and environment.

Can BIT certify a system?

No. BIT is developing assurance-oriented TEVV design and evidence planning; certification or approval remains with the authorized body.

Does every test require live operation?

No. Analysis, simulation and hardware-in-the-loop can resolve many questions before controlled live evaluation.

Sources and technical basis

  1. NIST Artificial Intelligence Risk Management Framework — Reference inclusion establishes technical context only; it does not claim conformity, approval or certification.
  2. EASA Artificial Intelligence Concept Paper, Issue 2
  3. U.S. GAO Artificial Intelligence Accountability Framework
  4. DoD Directive 3000.09 — Autonomy in Weapon Systems

Turn one claim into a testable evidence plan.

Define the mission baseline, scenario catalogue, pass criteria, instrumentation and evidence package for a scoped TEVV programme.

Request TEVV workshop

Unclassified · scoped workshops and pilot design