The yes or no decisions related to pilot intervention, continued flight, and safety of aircraft/persons guide the evaluator through a step-by-step process to one of three conditions (i.e., normal, abnormal, or emer- gency), nominally at three different levels as shown in Figure 3. The evalu- ator then is provided with another di- chotomous choice to indicate if the in- tervention required pilot action using an existing procedure from a pocket checklist or equivalent (e.g., Aircraft Flight Manual ) or otherwise an ad hoc procedure defined as either an adap- tation of an existing procedure or an entirely new creative procedure. The distinction between having to utilize an existing procedure versus an ad hoc procedure refines the PIRS numerical score within the PIRS Level where the PIRS Level determines the overall consequence avoided by the pilot’s intervention. The PIRS decision flow card also has provisions for jot- ting down a brief narrative identifying the AI system involved and the spe- cific pilot intervention. The resolution of any regulatory compliance issues discovered dur- ing the PIRS evaluation can then be categorized by PIRS rating and con- sequence, thereby providing a clear identification of where mitigations or additional testing may be re- quired. Based on the severity of the consequences avoided by the pilot’s intervention, the mitigations could entail improvements in the learned AI algorithms to implement/revise procedures for operating in known environments and foreseeable situ- ations and/or additional flight tests. These could provide the AI sys- tem additional learning opportuni- ties in uncertain environments and unforeseeable situations. However, a balanced evaluation also should identify any unwarranted/ incorrect pilot interventions. Hence, each pilot intervention must be vali- dated to determine if it was appropri- ate for maintaining safety margins or if it was inappropriate and thereby reduced safety margins.
It should be expected that DT&E and OT&E flights that involve AI systems will have flight testers onboard and/or remotely located.
T&E Flight Test The PIRS should be applied dur- ing developmental test and evalu- ation (DT&E) and operational test and evaluation (OT&E). The former validates the capability to operate within the intended operational mis- sion contexts, and the latter validates that the system under test can effec- tively execute its mission in a realis- tic operational environment that in- cludes an uncertain environment and unforeseen situations. It should be expected that DT&E and OT&E flights that involve AI sys- tems will have flight testers onboard and/or remotely located. The flight testers will determine if the AI sys- tems performed properly and if they are a reliable surrogate that demon- strates an equivalent (or better) level of performance and safety to that of a human pilot. Analogous to the other pilot rating scales, the PIRS does not involve a long and arduous or com- plicated process; hence, it is practical and easy to use in actual flight condi- tions. The Way Ahead AI systems on aircraft present unique and still unknown challenges for showing compliance with appli- cable regulations, standards, speci- fications, and guidance. However, human intervention rating scales, such as the PIRS, provide a viable method of evaluating and document- ing if an AI-driven system presents a hazard by ignoring, failing, or im- properly executing a task or function
that a human would have otherwise performed safely. JOSLIN is an associate professor at Embry- Riddle Aeronautical University where he de- veloped the capstone course for the M.S. Air- worthiness Engineering degree. He previously served as the Federal Aviation Administration (FFA) Chief Scientific and Technical Advisor for Flight Deck Technology and an FAA flight test pilot. Prior to that, he was a colonel in the U.S. Marine Corps, commander of DCMA Bell Helicopter, and a military test pilot at Naval Air Warfare Center (NAWC)-Aircraft Division Patuxent River, Md., and Naval Air Warfare Center-Weapons Division, China Lake, Calif. Joslin holds a Ph.D. in Aviation from Embry- Riddle, an M.S. in Aeronautical Engineering from the Naval Postgraduate School, and a B.S. in Mechanical Engineering from the Uni- versity of Florida. The author can be contacted at joslinr@erau.edu . The views expressed in this article are those of the author alone and not the Department of War. Reproduction or reposting of articles from Defense Acquisition magazine should credit the authors and the magazine.
Related Resources • Artificial Intelligence (AI) (Resource Hub) • Recommended Introductory Level AI Courses (Playlist) • SWE 0056 What is Artificial Intelligence (AI)? (Online Training Course) • Leveraging Artificial Intelligence (Webinar)
November-December 2025 | DEFENSE ACQUISITION | 27
Made with FlippingBook - Online Brochure Maker