NIST AI RMF: Mastering AI Risk Measurement and Trustworthiness

The National Institute of Standards and Technology (NIST) AI Resource Center (AIRC) offers a comprehensive suite of tools and guidance for organizations navigating the complexities of artificial intelligence. A critical component of responsible AI development and deployment lies within the 'Measure' function of the AI Risk Management Framework (RMF). This function is dedicated to establishing and applying robust methods for quantifying and assessing AI risks and their associated trustworthiness characteristics. In an era where AI systems are increasingly integrated into critical societal functions, the ability to accurately measure their performance, identify potential failures, and ensure their reliability is paramount. This section delves into the core principles and practices outlined by NIST for effective AI risk measurement, providing a roadmap for organizations to build and maintain trustworthy AI.

The development and utility of trustworthy AI systems are intrinsically linked to reliable measurements and evaluations of the underlying technologies and their application. Unlike traditional software, AI technologies present unique challenges. They can exhibit novel failure modes, possess an inherent dependence on training data that directly impacts data quality and representativeness, and are fundamentally socio-technical in nature. This means their behavior is influenced by societal dynamics, human interaction, and the broader context of their deployment. AI risks, as well as benefits, can emerge from the interplay of technical aspects with societal factors, including how a system is used, its interactions with other AI systems, who operates it, and the social environment in which it operates. Therefore, understanding 'what should be measured' is not a one-size-fits-all approach; it is contingent upon the specific purpose, audience, and evaluation needs of the AI system. These factors directly influence the selection of appropriate approaches and metrics for measuring AI risks that were enumerated during the 'Map' function of the AI RMF. As the AI landscape rapidly evolves, so too must the methods and metrics used for AI measurement to maintain their efficacy.

At its core, AI risk measurement is about establishing a systematic process for understanding and quantifying the potential harms and benefits associated with AI systems. This involves a proactive approach to identifying, tracking, and measuring known risks, errors, incidents, or negative impacts. A fundamental principle is the establishment of clear testing procedures and metrics designed to demonstrate whether the AI system is fit for its intended purpose and functioning as claimed. Furthermore, these evaluations must extend to assessing the AI system's trustworthiness, ensuring it operates reliably and ethically. Defining acceptable performance limits, such as acceptable error distributions, is crucial. When a system's performance deviates beyond these limits, clear suggestions for course correction must be in place. The competency of AI actors responsible for operating the system also requires definition and regular assessment. Transparency metrics are vital to ensure stakeholders have access to necessary information about the system's design, development, deployment, use, and evaluation. Similarly, accountability metrics help determine if AI designers, developers, and deployers maintain clear lines of responsibility and are open to inquiries. Finally, a robust measurement process includes documenting the criteria for metric selection and any considered but ultimately unused metrics, providing a transparent record of the evaluation strategy.

The selection of AI measurement approaches and metrics is a critical step guided by the risks identified during the 'Map' phase of the AI RMF. Organizations are encouraged to establish comprehensive approaches for detecting, tracking, and measuring known risks, errors, incidents, or negative impacts. This includes identifying specific testing procedures and metrics that can validate whether the AI system is fit for its intended purpose and performing as expected. Crucially, these procedures should also aim to demonstrate the AI system's trustworthiness. Defining acceptable thresholds for system performance, such as acceptable error rates, is essential, along with outlining suggested course correction actions when performance falls outside these limits. Beyond technical performance, the competency of AI actors involved in the system's operation must also be defined and regularly assessed through appropriate metrics. Transparency metrics are key to ensuring that stakeholders have access to essential information regarding the AI system's lifecycle, from design to evaluation. Likewise, accountability metrics help clarify and track the responsibilities of AI designers, developers, and deployers. A thorough documentation process should include the rationale behind metric selection and any metrics that were considered but not implemented, providing a clear audit trail for the measurement strategy.

When measuring AI risks, organizations should consider a range of metrics that capture different facets of system performance and trustworthiness. Key metrics include those that assess accuracy, reliability, fairness, and robustness. For instance, defining acceptable limits for system performance, such as the distribution of errors, is crucial. When a system's performance exceeds these limits, predefined course correction suggestions should be implemented. The competency of AI actors responsible for operating the system also needs to be measured and regularly assessed. Transparency metrics are vital for ensuring that stakeholders have access to necessary information about the system's design, development, deployment, use, and evaluation. Accountability metrics help determine whether AI designers, developers, and deployers maintain clear and transparent lines of responsibility and are open to inquiries. Furthermore, it is important to monitor AI system external inputs, including training data, models developed for other contexts, reused system components, and third-party tools and resources. Reporting these metrics can inform assessments of the system's generalizability and reliability. A critical aspect is to assess and document the system's performance both before and after deployment, accounting for existing and emergent risks. Finally, organizations must document any risks or trustworthiness characteristics identified in the 'Map' function that will not be measured, providing a clear justification for their exclusion from the measurement process.

Transparency and comprehensive documentation are cornerstones of effective AI risk measurement. Organizations should meticulously document how appropriate performance metrics, such as accuracy, will be monitored post-deployment. This includes detailing any corrective actions taken to enhance the quality, accuracy, reliability, and representativeness of the data used. Recommendations for data splits or evaluation measures, such as training, development, and testing sets, and metrics like accuracy or AUC, should be clearly outlined. If usability problems were addressed, documentation should confirm that user interfaces were tested for their intended purposes. The extent of testing conducted on the AI system to identify errors and limitations, including manual, automated, adversarial, and stress testing, must be recorded. AI transparency resources, such as GAO reports and ethics frameworks, can provide valuable guidance. Furthermore, organizations should document the criteria used for metric selection and include any considered but ultimately unused metrics. Monitoring AI system external inputs, including training data, models, components, and third-party tools, and reporting these metrics to inform assessments of generalizability and reliability are also essential. Documenting pre- vs. post-deployment system performance, including existing and emergent risks, is critical. Finally, any risks or trustworthiness characteristics identified during the 'Map' function that will not be measured must be documented with a clear justification for their non-measurement.

The dynamic nature of AI systems necessitates a continuous process of assessing and updating the appropriateness and effectiveness of measurement metrics and controls. Different AI tasks, such as neural networks or natural language processing, benefit from distinct evaluation techniques, and the specific use-case and operational settings significantly influence the suitability of these techniques. Factors like changes in operational settings and data drift or model drift highlight the importance of regularly reassessing and updating measurement metrics to enhance the reliability of AI system measurements. Organizations should assess the external validity of all measurements, understanding the degree to which findings from one context can generalize to others. The effectiveness of existing metrics and controls should be evaluated regularly throughout the AI system lifecycle. Reports of errors, incidents, and negative impacts must be documented, and the sufficiency and efficacy of existing metrics for repairs and upgrades should be assessed. When existing metrics prove insufficient or ineffective, new ones should be developed. Metrics to monitor, characterize, and track external inputs, including third-party tools, are also essential. Determining the frequency and scope for sharing metrics and related information with stakeholders and impacted communities is crucial. Utilizing stakeholder feedback processes, established in the 'Map' function, to capture, act upon, and share feedback from end-users and potentially impacted communities is vital. Collecting and reporting software quality metrics, such as bug occurrence rates, severity, time to response, and time to repair, further contributes to a comprehensive understanding of system performance.

Ensuring the comprehensive characterization of AI systems' performance and trustworthiness requires the involvement of diverse perspectives. Internal experts who were not directly involved in the front-line development of the system, and/or independent assessors, should participate in regular assessments and updates. Domain experts, users, AI actors external to the development and deployment teams, and affected communities should be consulted as necessary, based on the organization's risk tolerance. The AI RMF encourages evaluating Test, Evaluation, Validation, and Verification (TEVV) processes to identify risks and impacts effectively. Utilizing separate testing teams, as established in the 'Govern' function, can enable independent decision-making and course correction for AI systems, with processes and performance changes being tracked and documented. Prototyping AI systems with end-user populations early and continuously throughout the AI lifecycle is recommended, with test outcomes documented and course corrections implemented. The independence and stature of TEVV and oversight AI actors must be assessed to ensure they possess the necessary independence and resources for effective assurance, compliance, and feedback tasks. Evaluating interdisciplinary and demographically diverse internal teams, as established in 'Map 1.2', is also important. Furthermore, the effectiveness of external stakeholder feedback mechanisms, particularly for eliciting, evaluating, and integrating input from diverse groups, should be assessed. This includes evaluating how these mechanisms enhance AI actor visibility and decision-making regarding AI system risks and trustworthy characteristics. Finally, identifying and utilizing participatory approaches for assessing impacts that may arise from changes in system deployment, such as introducing new technology or decommissioning algorithms, is crucial.

A foundational element for building a valid and reliable AI measurement process is the meticulous documentation of all aspects related to testing, evaluation, validation, and verification (TEVV). This includes documenting the test sets used, the metrics applied, and detailed information about the tools employed during these processes. Such documentation fosters repeatability and consistency, which are essential for making informed AI risk management decisions. Organizations are encouraged to leverage existing industry best practices for transparency and documentation, which can include practices like 'datasheets for datasets' and 'model cards.' Regularly assessing the effectiveness of the tools used to document measurement approaches, test sets, metrics, processes, and materials is also recommended. As technology and best practices evolve, these tools should be updated accordingly to maintain their utility and relevance. The documentation should clearly articulate the purpose of the AI system and define an appropriate interval for checking its accuracy, bias, explainability, and other relevant characteristics. The extent to which the AI system's development, testing methodology, metrics, and performance outcomes have been documented should be clearly recorded. Resources such as the GAO's 'Artificial Intelligence: An Accountability Framework for Federal Agencies & Other Entities' and the WEF's 'Companion to the Model AI Governance Framework' can provide valuable guidance on best practices for documentation and transparency in AI evaluation.

When AI system evaluations involve human subjects or utilize data captured from human subjects, adherence to applicable requirements, particularly human subject protection, is non-negotiable. Federally funded research mandates the protection of human subjects, and this is a domain-specific requirement across various disciplines. Standard human subject protection procedures encompass safeguarding the welfare and interests of participants, designing evaluations to minimize risks, and ensuring completion of mandatory training on legal requirements and expectations. Evaluations of AI system performance that involve human subjects or their data must accurately reflect the intended population within the context of use. The use of non-representative data can lead to inaccurate assessments and potentially harmful outcomes. Recognizing the challenges in collecting data or performing evaluations that fully encompass an AI system's operational purview, organizations can connect human subject data collection and dataset practices directly to AI system contexts and purposes. This should be done in close collaboration with AI actors from the relevant domains. Organizations should follow established human subjects research requirements, including informed consent and compensation, during dataset collection activities. Analyzing differences between intended and actual populations of users or data subjects, including the likelihood of errors, incidents, or negative impacts, is also crucial. Utilizing disaggregated evaluation methods (e.g., by race, age, gender, ethnicity, ability, region) can significantly improve AI system performance when deployed in real-world settings. Establishing clear thresholds and alert procedures for dataset representativeness is a key step in ensuring ethical and effective AI evaluation.