Publication

Research Goal

The ability to assess a system is a necessity for repeatable and justifiable decision-making that is core to systems engineering. Assessment is a difficult task that has many approaches supporting it, such as value modeling, multi-objective optimization, and verification and validation. While arguments can be made for and against proposed approaches, these approaches at least give some repeatability and justification to decision-making concerning systems. Systems that incorporate AI have a reputation of being more difficult to assess compared to systems that do not incorporate AI. The uncertainty involved with the behavior of AI incorporated systems may change the applicability of assessment approaches used in non-AI incorporated systems. Furthermore, AI incorporated systems inherently integrate disciplines that may not have been needed in non-AI incorporated systems, such as Computer Science, Psychology, and Philosophy. The research goal of this work strives to form an interdisciplinary, comprehensive assessment framework for AI incorporated systems.

Objectives

In order to better understand how to assess an AI incorporated system, it is first important to understand what is believed to make such systems hard to assess. The perceived difficulty in assessing AI incorporated systems may be due to perceived system characteristics including evolving behaviors, immeasurable maintenance metrics, user acceptance and adoption problems, vagueness in system decision making, and challenges in integrating with legacy systems. The objective of this work is to focus on three perceived characteristics of AI incorporated systems that impact the ability for assessment. Specifically, the objectives are:

1. Define reliability and identify appropriate measures;
2. Form techniques to explain AI decisions and the decision-making process to users; and
3. Identify sources of emergent behaviors in AI incorporated systems.

Methodology

To accomplish Objective A (Reliability) the researchers have performed a cross-discipline literature review and are currently performing a survey of practitioners and experts. This evidence will be used to synthesize a definition of reliability and associated measures.

To accomplish Objective B (Explainable AI) the researchers have performed a literature review on explainable AI, interpretable ML, transparent ML, and comprehensible ML and are currently developing a user study to research display methods and types of information for transparency. This evidence will be used to synthesize an interdisciplinary technique to simplify AI/ML model interpretation.

To accomplish Objective C (Emergent behavior) the researchers are currently performing a multidisciplinary literature review to identify emergent behavior origins.

Expected Outcomes

This on-going research is expected to support the formation of a comprehensive assessment framework for AI incorporated systems. Specifically, this on-going research will produce a reliability definition, measures for reliability, techniques to explain AI processes, and identification of origins of emergent behavior in AI incorporated systems.

Broader Applications

Assessment of AI incorporated systems is necessary for systems engineering and most decision-making during the design process. An assessment framework that is based on evidence to address key challenges in AI incorporated systems would likely move closer to being repeatable and justifiable. Such an assessment framework removes biases from stakeholders, resulting in decisions that can be argued less on opinions and more on evidence. This is the trend that non-AI incorporated systems are heading, and this research enables AI-incorporated systems to follow a similar path.

Relevance and Significance

This research performs fundamental research on addressing challenges perceived to exist for AI systems. This work establishes a basis for future research on assessment frameworks. This research also provides insights on the validity of specific perceptions about AI incorporated systems.

Relationship to Conference Theme

A good AI assessment framework will inherently incorporate the ability to “balance” opportunity and risk, while optimizing the task to be accomplished by the system. This research focuses on key aspects to enable such an assessment. This research is most relevant to the conference tracks of Human-AI Teaming, Trustworthy AI, and SE4AI. The research assumes Human-AI teaming scenarios, and also specifically addresses Human-AI teaming in the research on explainable AI with user studies on display methods. The explainable AI research also addresses Trustworthy AI, by performing user studies on display methods for AI decisions and processes. An assessment framework for AI incorporated systems enables repeatable and justifiable systems engineering for AI systems.

Ethical Considerations

The research involving humans has been approved by the UAH IRB.