One Small Step for Space Acquisition Doctrine
Prompts had to be developed and tested through numerous trials until output and format were as consistent as possible across both models. Once prompts were finalized, the same prompt was used for both model inputs, and the output was noted. Both models analyzed codes, sentiment, and initial tenet alignment. Details on prompt engineering will be discussed later in this section. Next, the top 10 codes from ChatGPT and Claude were combined through a theme discovery process. The analysis considered the rank- ordered nature of each list and compared and combined them into a final list of five themes. The themes were then analyzed against their respective tenet—in conjunction with sentiment and alignment data—to conclude whether the survey results agreed with the spirit of the tenet (triangulation). For Questions 17–20, which did not reference a tenet, the top 5 list was compared to sentiment only. In all analyses, automatic data coding and themes from NVivo were compared as a final verification. Additionally, survey data were reviewed manually to show a correlation between survey responses and the tenets or other best practices. Coding with NVivo NVivo is a popular qualitative data analysis software tool used for social sciences, marketing, tourism, and forensics research. For this research, NVivo was used due to its popularity in the field and to verify or corroborate data discovered through detailed coding from the LLMs. The software connected directly to Qualtrics to import all participant survey data. Once the data were downloaded, a question-by-question coding analysis was conducted—similar to the method used for each LLM. Codes were autogenerated rapidly and automatically output into code themes.
152
Defense ARJ, Summer 2025, Vol. 32 No. 2: 132—193
Made with FlippingBook - Online Brochure Maker