Executive summary
The executive summary above outlines the purpose of this deep dive. It shows how to apply the NIST AI Risk Management Framework (AI RMF) Measure function to evaluate AI systems, how observability bridges testing and production, and why a paper’s headline score does not replace rigorous, documented measurement. The piece also clarifies which aspects of trustworthiness remain unevaluated and therefore must be explicitly recorded.
Introduction
Artificial intelligence (AI) systems are increasingly embedded in critical business processes. Decision makers often encounter research papers, pre‑prints, or vendor white‑papers that report impressive benchmark numbers. Those numbers can be tempting, but they rarely convey the full picture of risk, reliability, or the propensity to hallucinate. The National Institute of Standards and Technology (NIST) provides a structured approach to risk management through the Artificial Intelligence Risk Management Framework (AI RMF) 1.0. Within the framework, the Measure function defines how organizations should test, evaluate, benchmark, and monitor AI systems throughout their lifecycle. This article walks through the practical steps of applying Measure, explains the technical foundations of accuracy, reliability, and hallucination, and shows how observability turns offline tests into ongoing assurance.
Technical fundamentals
The AI RMF treats validity, reliability, accuracy, robustness, and misinformation as distinct trustworthy characteristics. NIST adopts the ISO 9000:2015 definition of validation – confirmation by objective evidence that requirements for a specific intended use have been fulfilled – and ISO/IEC TS 5723:2022 for reliability – the ability of an item to perform as required, without failure, for a given time interval, under given conditions [AI RMF Core]. Accuracy is defined as the closeness of results to true values or to values accepted as true, and it must be measured with realistic test sets and documented test methods. Robustness (or generalizability) captures performance stability when the system encounters circumstances that were not anticipated during development. Finally, the OWASP Top 10 for large‑language‑model (LLM) applications lists misinformation as a core vulnerability, describing it as the production of false content [OWASP].
These definitions are intentionally qualitative; the framework does not prescribe a single numeric threshold such as a percent‑level accuracy. Instead, it requires organizations to select appropriate metrics for the most significant risks, document any unmeasured properties, and maintain evidence of testing before deployment and during operation.
Architecture and components
The Measure function sits between Map and Manage in the AI RMF core. Map identifies intended use, risk scenarios, and trustworthiness characteristics. Measure then applies quantitative, qualitative, or mixed‑method tools to evaluate those characteristics. The outputs – test reports, uncertainty estimates, benchmark comparisons – feed Manage, which decides whether to keep, recalibrate, mitigate, or retire the system.
A minimal architecture for measuring and observing an AI service includes:
- Risk mapping layer – captures intended use, user personas, and mapped risks such as misinformation.
- Test harness – executes realistic test sets, records false‑positive/false‑negative rates, and logs uncertainty.
- Observability pipeline – streams production inputs and outputs to a monitoring service, compares live performance against the baseline established in testing, and raises alerts when drift is detected.
- Governance dashboard – aggregates measurement artifacts, stores evidence, and supports audit trails required by the AI RMF.
Detailed implementation approach
Below is a step‑by‑step approach that aligns with Measure 1 and Measure 1.1:
- Identify high‑impact risks – Use the Map output to rank risks by potential harm. For a conversational assistant, misinformation may be the top risk.
- Select metrics – Choose metrics that directly address the ranked risks. For misinformation, record the proportion of generated statements that contradict a vetted reference set; for reliability, record mean time between failures (MTBF) under defined load.
- Document unmeasurable properties – If a characteristic such as “ethical nuance” cannot be quantified, note it explicitly in the measurement plan.
- Assemble realistic test data – Curate a test set that reflects the operational domain, includes edge cases, and is annotated with ground‑truth values.
- Run offline evaluation – Execute the test harness, compute false‑positive/false‑negative rates, confidence intervals, and any domain‑specific scores. Record the methods used.
- Benchmark where appropriate – Compare results to publicly available benchmarks only when the benchmark’s task, data, and evaluation protocol are fully described.
- Establish monitoring cadence – Define a schedule (e.g., daily drift checks, weekly reliability summaries) for feeding live data into the observability pipeline.
- Create escalation path – Ensure a human can intervene when the system produces an untrusted output or when the monitoring alerts cross a predefined threshold.
- Iterate – Use the Manage function to recalibrate models, update test sets, or retire the system based on measurement evidence.
Example configuration snippet
model: gpt‑4
benchmark: truthfulqa
test_set: /data/truthfulqa_test.jsonl
metrics:
- name: false_positive_rate
threshold: 0.10
- name: false_negative_rate
threshold: 0.15
report_path: ./reports/measure_run_2026-11-12.json
The snippet illustrates that thresholds are part of the organization’s risk‑based decision, not a universal requirement.
Architecture and data‑flow
The following diagram visualizes the flow from risk mapping through measurement to management decisions. It is presented as a simple text‑based diagram suitable for inclusion in documentation.
[Map] --> identifies risks --> [Measure] --> runs test harness --> generates report -->
| |
| v
| [Observability] --> monitors live traffic --> alerts
| |
+---------------------------------+-------------------+
v
[Manage] --> keep / recalibrate / retire
The diagram emphasizes that misinformation is a risk identified in Map, not a floating metric.
Security and governance
Applying Measure requires controls that protect test data, model artifacts, and monitoring logs. NIST’s AI RMF stresses rights‑preserving practices; therefore, any personal data used in test sets must be de‑identified or used under explicit consent. Access to measurement reports should be limited to roles involved in risk governance, and audit logs must capture who created, modified, or approved each artifact. When integrating observability pipelines, ensure that data‑in‑transit encryption and role‑based access controls are in place to prevent leakage of sensitive model outputs.
Enterprise use cases
- Customer support chatbot – Map flags misinformation as high risk. Measure uses a curated FAQ test set, records false‑positive rates, and monitors live chat logs for deviation. Manage may disable the bot if drift exceeds a tolerance.
- Predictive maintenance model – Reliability is the primary characteristic. Measure records MTBF during staged rollout; observability tracks prediction accuracy against sensor readings. Manage schedules model retraining when reliability degrades.
- Legal document summarizer – Accuracy and validity are critical. Measure employs a benchmark of annotated legal excerpts, and a human‑in‑the‑loop review step is documented as an unmeasurable but mitigated risk.
Benefits and limitations
Benefits
- Provides repeatable, evidence‑based evaluation rather than relying on single benchmark scores.
- Aligns testing with the organization’s actual risk profile, ensuring resources focus on the most harmful failures.
- Creates a clear audit trail for regulators or internal auditors.
Limitations
- Some trustworthiness attributes, such as nuanced ethical reasoning, may remain unevaluated; the framework requires explicit documentation of these gaps.
- Observability for open‑ended language generation is less straightforward than for scalar predictions; it often relies on human review or custom rubrics.
- The process demands significant engineering effort to build test harnesses, monitoring pipelines, and governance dashboards.
Operational considerations
- Test set maintenance – Real‑world domains evolve; test data must be refreshed regularly to stay representative.
- Alert fatigue – Thresholds for drift or reliability alerts should be tuned to avoid excessive false alarms, which can erode trust in the monitoring system.
- Resource budgeting – Running large‑scale offline evaluations and continuous observability can be compute‑intensive; organizations should allocate capacity accordingly.
- Human‑in‑the‑loop workflows – For high‑impact decisions, the Manage function should include clear escalation paths to subject‑matter experts.
Best practices
- Document everything – From the rationale for metric selection to the exact version of the test set, every decision must be recorded.
- Start with the most significant risks – Measure 1.1 directs organizations to prioritize risks identified in Map.
- Pair quantitative metrics with qualitative review – Numbers alone cannot capture all failure modes; incorporate expert assessment where needed.
- Separate offline evaluation from production monitoring – Use the term observability to refer specifically to the ongoing, operational view of performance.
- Regularly revisit the measurement plan – As the system or its environment changes, update the test methodology and risk ranking.
Analysis
The sources collectively establish a risk‑based, evidence‑driven approach to AI evaluation. NIST’s AI RMF provides the high‑level structure (Govern, Map, Measure, Manage) and the detailed expectations for the Measure function, including the need for documented methods, uncertainty quantification, and continuous monitoring. The OWASP LLM risk list confirms that misinformation is a recognized vulnerability but does not prescribe a universal hallucination rate. Consequently, any claim of a single “hallucination percentage” would be unsupported. The Google MLOps description of A/B testing for predictive services illustrates the shape of production observation but does not directly translate to LLM evaluation, reinforcing the need for domain‑specific observability designs.
What the sources do not settle
- Universal thresholds – No source defines a numeric cut‑off that determines when a model is “acceptable” for any risk category.
- Exact methods for detecting hallucinations in open‑ended text – The framework requires a rubric and reference set, but the specific algorithm or tool is left to the organization.
- Frequency of re‑evaluation – While Measure calls for testing before deployment and regular monitoring, it does not prescribe an exact interval.
- Effectiveness of human oversight – The sources acknowledge the need for human intervention when the system cannot self‑correct, yet they provide no metric for measuring that oversight’s success.
Conclusion
Applying the NIST AI RMF Measure function turns vague benchmark scores into concrete, repeatable evidence of an AI system’s accuracy, reliability, and susceptibility to misinformation. By mapping risks, selecting appropriate metrics, documenting unmeasured properties, and establishing an observability pipeline, enterprises can maintain trustworthy AI services throughout their lifecycle. The framework’s strength lies in its flexibility: organizations decide which risks matter most, which metrics best capture those risks, and how often to reassess. At the same time, the framework explicitly highlights what cannot be measured, ensuring that decision makers remain aware of residual uncertainty.
References
These published CloudExchangers articles are further desk reading. They are not sources for the technical claims above.